2026-10-27 18:42:34,747 [ 216223 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy (runner:42, check_args_and_update_paths)
2026-10-27 18:42:34,747 [ 216223 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration (runner:90, check_args_and_update_paths)
2026-10-27 18:42:34,747 [ 216223 ] INFO : src dir is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy/src (runner:97, check_args_and_update_paths)
2026-10-27 18:42:34,747 [ 216223 ] INFO : base_configs_dir: /home/ubuntu/_work/_temp/test/git-repo-copy/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration  (runner:99, check_args_and_update_paths)
2026-10-27 18:42:34,747 [ 216223 ] INFO : Unknown image altinityinfra/dotnet-client (runner:357, <module>)
Running pytest container as: 'docker run   --rm --name clickhouse_integration_tests_4vajc8 --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/src/Server/grpc_protos:/ClickHouse/src/Server/grpc_protos --volume=/run:/run/host:ro --mount type=bind,source=/home/ubuntu/_work/_temp/test/dockerd_volume_dir,target=/var/lib/docker -e DOCKER_HELPER_TAG=1864 -e DOCKER_BASE_TAG=1864 -e DOCKER_KERBERIZED_HADOOP_TAG=1864 -e DOCKER_KERBEROS_KDC_TAG=1864 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=1864 -e DOCKER_MYSQL_JAVA_CLIENT_TAG=1864 -e DOCKER_MYSQL_JS_CLIENT_TAG=1864 -e DOCKER_MYSQL_PHP_CLIENT_TAG=1864 -e DOCKER_NGINX_DAV_TAG=1864 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=1864   -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_NEW_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_role/test.py::test_admin_option test_role/test.py::test_changing_default_roles_affects_new_sessions_only test_role/test.py::test_combine_privileges test_role/test.py::test_create_role test_role/test.py::test_function_current_roles test_role/test.py::test_grant_role_to_role test_role/test.py::test_introspection test_role/test.py::test_revoke_requires_admin_option 'test_role/test.py::test_role_expiration[False]' 'test_role/test.py::test_role_expiration[True]' test_role/test.py::test_roles_cache test_role/test.py::test_set_role test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts test_select_access_rights/test_main.py::test_alias test_select_access_rights/test_main.py::test_alias_columns test_select_access_rights/test_main.py::test_materialized_columns test_select_access_rights/test_main.py::test_select_all_columns test_select_access_rights/test_main.py::test_select_all_columns_with_table_grant test_select_access_rights/test_main.py::test_select_count test_select_access_rights/test_main.py::test_select_join test_select_access_rights/test_main.py::test_select_prewhere test_select_access_rights/test_main.py::test_select_single_column test_select_access_rights/test_main.py::test_select_single_column_with_table_grant test_select_access_rights/test_main.py::test_select_union test_select_access_rights/test_main.py::test_select_where test_select_access_rights/test_main.py::test_select_with_row_policy 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6]' test_settings_profile/test.py::test_allow_ddl test_settings_profile/test.py::test_allow_introspection test_settings_profile/test.py::test_alter_and_drop test_settings_profile/test.py::test_changeable_in_readonly test_settings_profile/test.py::test_changing_default_profiles_affects_new_sessions_only test_settings_profile/test.py::test_function_current_profiles test_settings_profile/test.py::test_inheritance test_settings_profile/test.py::test_set_profile test_settings_profile/test.py::test_settings_aliases test_settings_profile/test.py::test_settings_from_granted_role test_settings_profile/test.py::test_show_profiles test_settings_profile/test.py::test_smoke test_sql_user_defined_functions_on_cluster/test.py::test_sql_user_defined_functions_on_cluster test_storage_delta/test.py::test_checkpoint test_storage_delta/test.py::test_metadata test_storage_delta/test.py::test_multiple_log_files test_storage_delta/test.py::test_partition_by test_storage_delta/test.py::test_single_log_file test_storage_delta/test.py::test_types test_storage_kerberized_kafka/test.py::test_kafka_json_as_string test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_no_kdc test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_request_new_ticket_after_expiration 'test_storage_meilisearch/test.py::test_incorrect_data_type[False]' 'test_storage_meilisearch/test.py::test_incorrect_data_type_secure[False]' 'test_storage_meilisearch/test.py::test_insert[False]' 'test_storage_meilisearch/test.py::test_insert_secure[False]' 'test_storage_meilisearch/test.py::test_meilimatch[False]' 'test_storage_meilisearch/test.py::test_meilimatch_secure[False]' 'test_storage_meilisearch/test.py::test_named_collection[False]' 'test_storage_meilisearch/test.py::test_named_collection_secure[False]' 'test_storage_meilisearch/test.py::test_security_levels[False]' 'test_storage_meilisearch/test.py::test_simple_select[False]' 'test_storage_meilisearch/test.py::test_simple_select_secure[False]' 'test_storage_meilisearch/test.py::test_table_function[False]' 'test_storage_meilisearch/test.py::test_table_function_secure[False]' 'test_storage_meilisearch/test.py::test_types[False]' 'test_storage_meilisearch/test.py::test_types_in_table_function[False]' 'test_storage_mongodb/test.py::test_arrays[False]' 'test_storage_mongodb/test.py::test_auth_source[False]' 'test_storage_mongodb/test.py::test_complex_data_type[False]' 'test_storage_mongodb/test.py::test_incorrect_data_type[False]' 'test_storage_mongodb/test.py::test_missing_columns[False]' 'test_storage_mongodb/test.py::test_no_credentials[False]' 'test_storage_mongodb/test.py::test_predefined_connection_configuration[False]' 'test_storage_mongodb/test.py::test_secure_connection[True]' 'test_storage_mongodb/test.py::test_simple_insert_select[False]' 'test_storage_mongodb/test.py::test_simple_select[False]' 'test_storage_mongodb/test.py::test_simple_select_from_view[False]' test_storage_s3/test.py::test_ast_auth_headers test_storage_s3/test.py::test_create_new_files_on_insert test_storage_s3/test.py::test_custom_auth_headers test_storage_s3/test.py::test_custom_auth_headers_exclusion test_storage_s3/test.py::test_empty_file 'test_storage_s3/test.py::test_empty_put[minio]' test_storage_s3/test.py::test_environment_credentials test_storage_s3/test.py::test_filtering_by_file_or_path test_storage_s3/test.py::test_format_detection 'test_storage_s3/test.py::test_get_file_with_special[plus]' 'test_storage_s3/test.py::test_get_file_with_special[space]' 'test_storage_s3/test.py::test_get_path_with_special[plus2]' 'test_storage_s3/test.py::test_get_path_with_special[plus]' 'test_storage_s3/test.py::test_get_path_with_special[space]' test_storage_s3/test.py::test_infinite_redirect test_storage_s3/test.py::test_insert_select_schema_inference test_storage_s3/test.py::test_insert_with_path_with_globs 'test_storage_s3/test.py::test_multipart[negative]'  -vvv" altinityinfra/integration-tests-runner:1864 '.
Start tests
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.7.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /ClickHouse/tests/integration
configfile: pytest.ini
plugins: xdist-3.5.0, random-0.2, repeat-0.9.4, order-1.0.1, timeout-2.2.0
timeout: 900.0s
timeout method: signal
timeout func_only: False
created: 10/10 workers
10 workers [100 items]

scheduling tests via LoadFileScheduling

test_select_access_rights/test_main.py::test_alias 
test_storage_mongodb/test.py::test_arrays[False] 
test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1] 
test_role/test.py::test_admin_option 
test_storage_kerberized_kafka/test.py::test_kafka_json_as_string 
test_settings_profile/test.py::test_allow_ddl 
test_storage_delta/test.py::test_checkpoint 
test_storage_meilisearch/test.py::test_incorrect_data_type[False] 
test_storage_s3/test.py::test_ast_auth_headers 
test_s3_table_functions/test.py::test_s3_table_functions 
[gw4] [  1%] PASSED test_settings_profile/test.py::test_allow_ddl 
[gw2] [  2%] PASSED test_select_access_rights/test_main.py::test_alias 
test_select_access_rights/test_main.py::test_alias_columns 
test_settings_profile/test.py::test_allow_introspection 
[gw3] [  3%] PASSED test_role/test.py::test_admin_option 
test_role/test.py::test_changing_default_roles_affects_new_sessions_only 
[gw3] [  4%] PASSED test_role/test.py::test_changing_default_roles_affects_new_sessions_only 
[gw1] [  5%] PASSED test_storage_meilisearch/test.py::test_incorrect_data_type[False] 
test_storage_meilisearch/test.py::test_incorrect_data_type_secure[False] 
[gw2] [  6%] FAILED test_select_access_rights/test_main.py::test_alias_columns 
test_role/test.py::test_combine_privileges 
test_select_access_rights/test_main.py::test_materialized_columns 
[gw1] [  7%] PASSED test_storage_meilisearch/test.py::test_incorrect_data_type_secure[False] 
test_storage_meilisearch/test.py::test_insert[False] 
[gw4] [  8%] PASSED test_settings_profile/test.py::test_allow_introspection 
[gw3] [  9%] PASSED test_role/test.py::test_combine_privileges 
[gw2] [ 10%] PASSED test_select_access_rights/test_main.py::test_materialized_columns 
test_settings_profile/test.py::test_alter_and_drop 
test_role/test.py::test_create_role 
test_select_access_rights/test_main.py::test_select_all_columns 
[gw3] [ 11%] PASSED test_role/test.py::test_create_role 
[gw4] [ 12%] PASSED test_settings_profile/test.py::test_alter_and_drop 
[gw2] [ 13%] PASSED test_select_access_rights/test_main.py::test_select_all_columns 
test_role/test.py::test_function_current_roles 
[gw5] [ 14%] PASSED test_storage_mongodb/test.py::test_arrays[False] 
test_storage_mongodb/test.py::test_auth_source[False] 
test_select_access_rights/test_main.py::test_select_all_columns_with_table_grant 
test_settings_profile/test.py::test_changeable_in_readonly 
[gw5] [ 15%] PASSED test_storage_mongodb/test.py::test_auth_source[False] 
test_storage_mongodb/test.py::test_complex_data_type[False] 
[gw7] [ 16%] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1] 
test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2] 
[gw2] [ 17%] PASSED test_select_access_rights/test_main.py::test_select_all_columns_with_table_grant 
[gw3] [ 18%] PASSED test_role/test.py::test_function_current_roles 
test_select_access_rights/test_main.py::test_select_count 
[gw4] [ 19%] PASSED test_settings_profile/test.py::test_changeable_in_readonly 
test_role/test.py::test_grant_role_to_role 
[gw5] [ 20%] PASSED test_storage_mongodb/test.py::test_complex_data_type[False] 
test_storage_mongodb/test.py::test_incorrect_data_type[False] 
test_settings_profile/test.py::test_changing_default_profiles_affects_new_sessions_only 
[gw2] [ 21%] FAILED test_select_access_rights/test_main.py::test_select_count 
test_select_access_rights/test_main.py::test_select_join 
[gw4] [ 22%] PASSED test_settings_profile/test.py::test_changing_default_profiles_affects_new_sessions_only 
[gw5] [ 23%] PASSED test_storage_mongodb/test.py::test_incorrect_data_type[False] 
test_storage_mongodb/test.py::test_missing_columns[False] 
test_settings_profile/test.py::test_function_current_profiles 
[gw2] [ 24%] FAILED test_select_access_rights/test_main.py::test_select_join 
[gw3] [ 25%] PASSED test_role/test.py::test_grant_role_to_role 
[gw7] [ 26%] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2] 
test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3] 
test_select_access_rights/test_main.py::test_select_prewhere 
test_role/test.py::test_introspection 
[gw5] [ 27%] PASSED test_storage_mongodb/test.py::test_missing_columns[False] 
test_storage_mongodb/test.py::test_no_credentials[False] 
[gw5] [ 28%] PASSED test_storage_mongodb/test.py::test_no_credentials[False] 
test_storage_mongodb/test.py::test_predefined_connection_configuration[False] 
[gw1] [ 29%] PASSED test_storage_meilisearch/test.py::test_insert[False] 
test_storage_meilisearch/test.py::test_insert_secure[False] 
[gw5] [ 30%] PASSED test_storage_mongodb/test.py::test_predefined_connection_configuration[False] 
test_storage_mongodb/test.py::test_secure_connection[True] 
[gw4] [ 31%] PASSED test_settings_profile/test.py::test_function_current_profiles 
test_settings_profile/test.py::test_inheritance 
[gw7] [ 32%] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3] 
test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4] 
[gw2] [ 33%] PASSED test_select_access_rights/test_main.py::test_select_prewhere 
test_select_access_rights/test_main.py::test_select_single_column 
[gw9] [ 34%] PASSED test_storage_kerberized_kafka/test.py::test_kafka_json_as_string 
test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_no_kdc 
[gw4] [ 35%] PASSED test_settings_profile/test.py::test_inheritance 
[gw2] [ 36%] PASSED test_select_access_rights/test_main.py::test_select_single_column 
test_select_access_rights/test_main.py::test_select_single_column_with_table_grant 
test_settings_profile/test.py::test_set_profile 
[gw7] [ 37%] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4] 
test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5] 
[gw4] [ 38%] PASSED test_settings_profile/test.py::test_set_profile 
test_settings_profile/test.py::test_settings_aliases 
[gw2] [ 39%] PASSED test_select_access_rights/test_main.py::test_select_single_column_with_table_grant 
test_select_access_rights/test_main.py::test_select_union 
[gw4] [ 40%] PASSED test_settings_profile/test.py::test_settings_aliases 
test_settings_profile/test.py::test_settings_from_granted_role 
[gw3] [ 41%] PASSED test_role/test.py::test_introspection 
[gw7] [ 42%] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5] 
test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6] 
test_role/test.py::test_revoke_requires_admin_option 
[gw2] [ 43%] PASSED test_select_access_rights/test_main.py::test_select_union 
test_select_access_rights/test_main.py::test_select_where 
[gw1] [ 44%] PASSED test_storage_meilisearch/test.py::test_insert_secure[False] 
test_storage_meilisearch/test.py::test_meilimatch[False] 
[gw7] [ 45%] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6] 
[gw2] [ 46%] PASSED test_select_access_rights/test_main.py::test_select_where 
test_select_access_rights/test_main.py::test_select_with_row_policy 
[gw4] [ 47%] PASSED test_settings_profile/test.py::test_settings_from_granted_role 
test_settings_profile/test.py::test_show_profiles 
[gw3] [ 48%] PASSED test_role/test.py::test_revoke_requires_admin_option 
[gw2] [ 49%] PASSED test_select_access_rights/test_main.py::test_select_with_row_policy 
test_role/test.py::test_role_expiration[False] 
[gw4] [ 50%] FAILED test_settings_profile/test.py::test_show_profiles 
test_settings_profile/test.py::test_smoke 
[gw4] [ 51%] PASSED test_settings_profile/test.py::test_smoke 
[gw3] [ 52%] PASSED test_role/test.py::test_role_expiration[False] 
test_role/test.py::test_role_expiration[True] 
[gw1] [ 53%] PASSED test_storage_meilisearch/test.py::test_meilimatch[False] 
test_storage_meilisearch/test.py::test_meilimatch_secure[False] 
[gw3] [ 54%] PASSED test_role/test.py::test_role_expiration[True] 
test_role/test.py::test_roles_cache 
[gw8] [ 55%] PASSED test_s3_table_functions/test.py::test_s3_table_functions 
test_s3_table_functions/test.py::test_s3_table_functions_timeouts 
[gw1] [ 56%] PASSED test_storage_meilisearch/test.py::test_meilimatch_secure[False] 
test_storage_meilisearch/test.py::test_named_collection[False] 
[gw1] [ 57%] PASSED test_storage_meilisearch/test.py::test_named_collection[False] 
test_storage_meilisearch/test.py::test_named_collection_secure[False] 
[gw1] [ 58%] PASSED test_storage_meilisearch/test.py::test_named_collection_secure[False] 
test_storage_meilisearch/test.py::test_security_levels[False] 
[gw1] [ 59%] PASSED test_storage_meilisearch/test.py::test_security_levels[False] 
test_storage_meilisearch/test.py::test_simple_select[False] 
[gw3] [ 60%] PASSED test_role/test.py::test_roles_cache 
test_role/test.py::test_set_role 
[gw3] [ 61%] PASSED test_role/test.py::test_set_role 
[gw1] [ 62%] PASSED test_storage_meilisearch/test.py::test_simple_select[False] 
test_storage_meilisearch/test.py::test_simple_select_secure[False] 
[gw1] [ 63%] PASSED test_storage_meilisearch/test.py::test_simple_select_secure[False] 
test_storage_meilisearch/test.py::test_table_function[False] 
[gw1] [ 64%] PASSED test_storage_meilisearch/test.py::test_table_function[False] 
test_storage_meilisearch/test.py::test_table_function_secure[False] 
[gw1] [ 65%] PASSED test_storage_meilisearch/test.py::test_table_function_secure[False] 
test_storage_meilisearch/test.py::test_types[False] 
[gw5] [ 66%] FAILED test_storage_mongodb/test.py::test_secure_connection[True] 
test_storage_mongodb/test.py::test_simple_insert_select[False] 
[gw1] [ 67%] PASSED test_storage_meilisearch/test.py::test_types[False] 
test_storage_meilisearch/test.py::test_types_in_table_function[False] 
[gw6] [ 68%] PASSED test_storage_delta/test.py::test_checkpoint 
test_storage_delta/test.py::test_metadata 
[gw1] [ 69%] PASSED test_storage_meilisearch/test.py::test_types_in_table_function[False] 
[gw6] [ 70%] PASSED test_storage_delta/test.py::test_metadata 
test_storage_delta/test.py::test_multiple_log_files 
[gw5] [ 71%] PASSED test_storage_mongodb/test.py::test_simple_insert_select[False] 
test_storage_mongodb/test.py::test_simple_select[False] 
[gw5] [ 72%] PASSED test_storage_mongodb/test.py::test_simple_select[False] 
test_storage_mongodb/test.py::test_simple_select_from_view[False] 
[gw5] [ 73%] PASSED test_storage_mongodb/test.py::test_simple_select_from_view[False] 
[gw6] [ 74%] PASSED test_storage_delta/test.py::test_multiple_log_files 
test_storage_delta/test.py::test_partition_by 
[gw6] [ 75%] PASSED test_storage_delta/test.py::test_partition_by 
test_storage_delta/test.py::test_single_log_file 
[gw6] [ 76%] PASSED test_storage_delta/test.py::test_single_log_file 
test_storage_delta/test.py::test_types 
[gw6] [ 77%] PASSED test_storage_delta/test.py::test_types 
[gw8] [ 78%] PASSED test_s3_table_functions/test.py::test_s3_table_functions_timeouts 
[gw0] [ 79%] ERROR test_storage_s3/test.py::test_ast_auth_headers 
test_storage_s3/test.py::test_create_new_files_on_insert 
[gw0] [ 80%] ERROR test_storage_s3/test.py::test_create_new_files_on_insert 
test_storage_s3/test.py::test_custom_auth_headers 
[gw0] [ 81%] ERROR test_storage_s3/test.py::test_custom_auth_headers 
test_storage_s3/test.py::test_custom_auth_headers_exclusion 
[gw0] [ 82%] ERROR test_storage_s3/test.py::test_custom_auth_headers_exclusion 
test_storage_s3/test.py::test_empty_file 
[gw0] [ 83%] ERROR test_storage_s3/test.py::test_empty_file 
test_storage_s3/test.py::test_empty_put[minio] 
[gw0] [ 84%] ERROR test_storage_s3/test.py::test_empty_put[minio] 
test_storage_s3/test.py::test_environment_credentials 
[gw0] [ 85%] ERROR test_storage_s3/test.py::test_environment_credentials 
test_storage_s3/test.py::test_filtering_by_file_or_path 
[gw0] [ 86%] ERROR test_storage_s3/test.py::test_filtering_by_file_or_path 
test_storage_s3/test.py::test_format_detection 
[gw0] [ 87%] ERROR test_storage_s3/test.py::test_format_detection 
test_storage_s3/test.py::test_get_file_with_special[plus] 
[gw0] [ 88%] ERROR test_storage_s3/test.py::test_get_file_with_special[plus] 
test_storage_s3/test.py::test_get_file_with_special[space] 
[gw0] [ 89%] ERROR test_storage_s3/test.py::test_get_file_with_special[space] 
test_storage_s3/test.py::test_get_path_with_special[plus2] 
[gw0] [ 90%] ERROR test_storage_s3/test.py::test_get_path_with_special[plus2] 
test_storage_s3/test.py::test_get_path_with_special[plus] 
[gw0] [ 91%] ERROR test_storage_s3/test.py::test_get_path_with_special[plus] 
test_storage_s3/test.py::test_get_path_with_special[space] 
[gw0] [ 92%] ERROR test_storage_s3/test.py::test_get_path_with_special[space] 
test_storage_s3/test.py::test_infinite_redirect 
[gw0] [ 93%] ERROR test_storage_s3/test.py::test_infinite_redirect 
test_storage_s3/test.py::test_insert_select_schema_inference 
[gw0] [ 94%] ERROR test_storage_s3/test.py::test_insert_select_schema_inference 
test_storage_s3/test.py::test_insert_with_path_with_globs 
[gw0] [ 95%] ERROR test_storage_s3/test.py::test_insert_with_path_with_globs 
test_storage_s3/test.py::test_multipart[negative] 
[gw0] [ 96%] ERROR test_storage_s3/test.py::test_multipart[negative] 
[gw9] [ 97%] PASSED test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_no_kdc 
test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_request_new_ticket_after_expiration 
test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 
[gw8] [ 98%] PASSED test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 
[gw9] [ 99%] PASSED test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_request_new_ticket_after_expiration 
test_sql_user_defined_functions_on_cluster/test.py::test_sql_user_defined_functions_on_cluster 
[gw9] [100%] FAILED test_sql_user_defined_functions_on_cluster/test.py::test_sql_user_defined_functions_on_cluster 

==================================== ERRORS ====================================
___________________ ERROR at setup of test_ast_auth_headers ____________________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
---------------------------- Captured stdout setup -----------------------------
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
---------------------------- Captured stderr setup -----------------------------
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
------------------------------ Captured log setup ------------------------------
2026-10-27 17:42:38 [ 373 ] DEBUG : Command:['docker ps | wc -l'] (cluster.py:105, run_and_check)
2026-10-27 17:42:38 [ 373 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check)
2026-10-27 17:42:38 [ 373 ] DEBUG : No running containers (conftest.py:60, cleanup_environment)
2026-10-27 17:42:38 [ 373 ] DEBUG : Pruning Docker networks (conftest.py:62, cleanup_environment)
2026-10-27 17:42:38 [ 373 ] DEBUG : Command:['docker network prune --force'] (cluster.py:105, run_and_check)
2026-10-27 17:42:38 [ 373 ] DEBUG : Stderr:Error response from daemon: a prune operation is already running (cluster.py:115, run_and_check)
2026-10-27 17:42:38 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check)
2026-10-27 17:42:38 [ 373 ] DEBUG : Command:["sysctl net.ipv4.ip_local_port_range='55000 65535'"] (cluster.py:105, run_and_check)
2026-10-27 17:42:38 [ 373 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:113, run_and_check)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                  DOCKER_KERBEROS_KDC_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV         CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                             MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                        JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                             TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                                 HOSTNAME 02b61c2481ca (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                                    SHLVL 0 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                                     HOME /root (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                                   OLDPWD / (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                        DOCKER_HELPER_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                         PYTHONUNBUFFERED 1 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                            UBSAN_OPTIONS print_stacktrace=1 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                           PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_role/test.py::test_admin_option test_role/test.py::test_changing_default_roles_affects_new_sessions_only test_role/test.py::test_combine_privileges test_role/test.py::test_create_role test_role/test.py::test_function_current_roles test_role/test.py::test_grant_role_to_role test_role/test.py::test_introspection test_role/test.py::test_revoke_requires_admin_option 'test_role/test.py::test_role_expiration[False]' 'test_role/test.py::test_role_expiration[True]' test_role/test.py::test_roles_cache test_role/test.py::test_set_role test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts test_select_access_rights/test_main.py::test_alias test_select_access_rights/test_main.py::test_alias_columns test_select_access_rights/test_main.py::test_materialized_columns test_select_access_rights/test_main.py::test_select_all_columns test_select_access_rights/test_main.py::test_select_all_columns_with_table_grant test_select_access_rights/test_main.py::test_select_count test_select_access_rights/test_main.py::test_select_join test_select_access_rights/test_main.py::test_select_prewhere test_select_access_rights/test_main.py::test_select_single_column test_select_access_rights/test_main.py::test_select_single_column_with_table_grant test_select_access_rights/test_main.py::test_select_union test_select_access_rights/test_main.py::test_select_where test_select_access_rights/test_main.py::test_select_with_row_policy 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6]' test_settings_profile/test.py::test_allow_ddl test_settings_profile/test.py::test_allow_introspection test_settings_profile/test.py::test_alter_and_drop test_settings_profile/test.py::test_changeable_in_readonly test_settings_profile/test.py::test_changing_default_profiles_affects_new_sessions_only test_settings_profile/test.py::test_function_current_profiles test_settings_profile/test.py::test_inheritance test_settings_profile/test.py::test_set_profile test_settings_profile/test.py::test_settings_aliases test_settings_profile/test.py::test_settings_from_granted_role test_settings_profile/test.py::test_show_profiles test_settings_profile/test.py::test_smoke test_sql_user_defined_functions_on_cluster/test.py::test_sql_user_defined_functions_on_cluster test_storage_delta/test.py::test_checkpoint test_storage_delta/test.py::test_metadata test_storage_delta/test.py::test_multiple_log_files test_storage_delta/test.py::test_partition_by test_storage_delta/test.py::test_single_log_file test_storage_delta/test.py::test_types test_storage_kerberized_kafka/test.py::test_kafka_json_as_string test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_no_kdc test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_request_new_ticket_after_expiration 'test_storage_meilisearch/test.py::test_incorrect_data_type[False]' 'test_storage_meilisearch/test.py::test_incorrect_data_type_secure[False]' 'test_storage_meilisearch/test.py::test_insert[False]' 'test_storage_meilisearch/test.py::test_insert_secure[False]' 'test_storage_meilisearch/test.py::test_meilimatch[False]' 'test_storage_meilisearch/test.py::test_meilimatch_secure[False]' 'test_storage_meilisearch/test.py::test_named_collection[False]' 'test_storage_meilisearch/test.py::test_named_collection_secure[False]' 'test_storage_meilisearch/test.py::test_security_levels[False]' 'test_storage_meilisearch/test.py::test_simple_select[False]' 'test_storage_meilisearch/test.py::test_simple_select_secure[False]' 'test_storage_meilisearch/test.py::test_table_function[False]' 'test_storage_meilisearch/test.py::test_table_function_secure[False]' 'test_storage_meilisearch/test.py::test_types[False]' 'test_storage_meilisearch/test.py::test_types_in_table_function[False]' 'test_storage_mongodb/test.py::test_arrays[False]' 'test_storage_mongodb/test.py::test_auth_source[False]' 'test_storage_mongodb/test.py::test_complex_data_type[False]' 'test_storage_mongodb/test.py::test_incorrect_data_type[False]' 'test_storage_mongodb/test.py::test_missing_columns[False]' 'test_storage_mongodb/test.py::test_no_credentials[False]' 'test_storage_mongodb/test.py::test_predefined_connection_configuration[False]' 'test_storage_mongodb/test.py::test_secure_connection[True]' 'test_storage_mongodb/test.py::test_simple_insert_select[False]' 'test_storage_mongodb/test.py::test_simple_select[False]' 'test_storage_mongodb/test.py::test_simple_select_from_view[False]' test_storage_s3/test.py::test_ast_auth_headers test_storage_s3/test.py::test_create_new_files_on_insert test_storage_s3/test.py::test_custom_auth_headers test_storage_s3/test.py::test_custom_auth_headers_exclusion test_storage_s3/test.py::test_empty_file 'test_storage_s3/test.py::test_empty_put[minio]' test_storage_s3/test.py::test_environment_credentials test_storage_s3/test.py::test_filtering_by_file_or_path test_storage_s3/test.py::test_format_detection 'test_storage_s3/test.py::test_get_file_with_special[plus]' 'test_storage_s3/test.py::test_get_file_with_special[space]' 'test_storage_s3/test.py::test_get_path_with_special[plus2]' 'test_storage_s3/test.py::test_get_path_with_special[plus]' 'test_storage_s3/test.py::test_get_path_with_special[space]' test_storage_s3/test.py::test_infinite_redirect test_storage_s3/test.py::test_insert_select_schema_inference test_storage_s3/test.py::test_insert_with_path_with_globs 'test_storage_s3/test.py::test_multipart[negative]'  -vvv (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV    CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                     COMPOSE_HTTP_TIMEOUT 600 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV              DOCKER_MYSQL_PHP_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                 DOCKER_DOTNET_CLIENT_TAG latest (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV         CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV               DOCKER_MYSQL_JS_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                                     PATH /spark-3.3.2-bin-hadoop3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV             DOCKER_KERBERIZED_HADOOP_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                           DOCKER_CHANNEL stable (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                    DOCKER_CLIENT_TIMEOUT 300 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV        DOCKER_POSTGRESQL_JAVA_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                     DOCKER_NGINX_DAV_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV           DOCKER_MYSQL_GOLANG_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV              CLICKHOUSE_USE_NEW_ANALYZER 1 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                                      PWD /ClickHouse/tests/integration (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV             DOCKER_MYSQL_JAVA_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV       CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV         CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                                       TZ Etc/UTC (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                                JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                          DOCKER_BASE_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                               SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                                 LC_CTYPE C.UTF-8 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                 INTEGRATION_TESTS_RUN_ID 0 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                  PYTEST_XDIST_TESTRUNUID 3f31a6e98a9b464fb8e4acb1a75d554d (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                      PYTEST_XDIST_WORKER gw0 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                PYTEST_XDIST_WORKER_COUNT 10 (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : ENV                      PYTEST_CURRENT_TEST test_storage_s3/test.py::test_ast_auth_headers (setup) (cluster.py:343, __init__)
2026-10-27 17:42:38 [ 373 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:636, __init__)
2026-10-27 17:42:39 [ 373 ] DEBUG : Removed :/ClickHouse/tests/integration/test_storage_s3/_instances_0 (cluster.py:639, __init__)
2026-10-27 17:42:39 [ 373 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1629, add_instance)
2026-10-27 17:42:39 [ 373 ] DEBUG : Cluster name: project_name:rootteststorages3. Added instance name:restricted_dummy tag:1864 base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', '--project-name', 'rootteststorages3', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/compose (cluster.py:1915, add_instance)
2026-10-27 17:42:39 [ 373 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1629, add_instance)
2026-10-27 17:42:39 [ 373 ] DEBUG : Cluster name: project_name:rootteststorages3. Added instance name:dummy tag:1864 base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', '--project-name', 'rootteststorages3', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/docker-compose.yml'] docker_compose_yml_dir:/compose (cluster.py:1915, add_instance)
2026-10-27 17:42:39 [ 373 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1629, add_instance)
2026-10-27 17:42:39 [ 373 ] DEBUG : Cluster name: project_name:rootteststorages3. Added instance name:dummy_without_named_collections tag:1864 base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', '--project-name', 'rootteststorages3', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/docker-compose.yml'] docker_compose_yml_dir:/compose (cluster.py:1915, add_instance)
2026-10-27 17:42:39 [ 373 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1629, add_instance)
2026-10-27 17:42:39 [ 373 ] DEBUG : Cluster name: project_name:rootteststorages3. Added instance name:s3_max_redirects tag:1864 base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', '--project-name', 'rootteststorages3', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/docker-compose.yml'] docker_compose_yml_dir:/compose (cluster.py:1915, add_instance)
2026-10-27 17:42:39 [ 373 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1629, add_instance)
2026-10-27 17:42:39 [ 373 ] DEBUG : Cluster name: project_name:rootteststorages3. Added instance name:s3_non_default tag:1864 base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', '--project-name', 'rootteststorages3', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_non_default/docker-compose.yml'] docker_compose_yml_dir:/compose (cluster.py:1915, add_instance)
2026-10-27 17:42:39 [ 373 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1629, add_instance)
2026-10-27 17:42:39 [ 373 ] DEBUG : Cluster name: project_name:rootteststorages3. Added instance name:s3_with_environment_credentials tag:1864 base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', '--project-name', 'rootteststorages3', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_non_default/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_with_environment_credentials/docker-compose.yml'] docker_compose_yml_dir:/compose (cluster.py:1915, add_instance)
2026-10-27 17:42:39 [ 373 ] INFO : Starting cluster... (test.py:90, started_cluster)
2026-10-27 17:42:39 [ 373 ] INFO : Running tests in /ClickHouse/tests/integration/test_storage_s3/test.py (cluster.py:2637, start)
2026-10-27 17:42:39 [ 373 ] DEBUG : Cluster start called. is_up=False (cluster.py:2644, start)
2026-10-27 17:42:39 [ 373 ] DEBUG : !!! Docker info: Client: Docker Engine - Community
 Version:    29.9.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 38
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b7a25261e029d68978bbf290df0a62fc819b3c51
 runc version: v1.5.2-0-g29dd3dc2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-198-generic
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.33GiB
 Name: 02b61c2481ca
 ID: cbb4a713-cdad-4d08-9907-2668eda59ee0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  65.108.242.32:5000
  127.0.0.0/8
 Registry Mirrors:
  http://65.108.242.32:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 24

 (cluster.py:722, print_all_docker_pieces)
2026-10-27 17:42:39 [ 373 ] DEBUG : Docker networks for project rootteststorages3 are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:733, print_all_docker_pieces)
2026-10-27 17:42:39 [ 373 ] DEBUG : Docker containers for project rootteststorages3 are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:741, print_all_docker_pieces)
2026-10-27 17:42:39 [ 373 ] DEBUG : Docker volumes for project rootteststorages3 are DRIVER    VOLUME NAME
 (cluster.py:749, print_all_docker_pieces)
2026-10-27 17:42:39 [ 373 ] DEBUG : Cleanup called (cluster.py:754, cleanup)
2026-10-27 17:42:39 [ 373 ] DEBUG : !!! Docker info: Client: Docker Engine - Community
 Version:    29.9.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 38
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b7a25261e029d68978bbf290df0a62fc819b3c51
 runc version: v1.5.2-0-g29dd3dc2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-198-generic
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.33GiB
 Name: 02b61c2481ca
 ID: cbb4a713-cdad-4d08-9907-2668eda59ee0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  65.108.242.32:5000
  127.0.0.0/8
 Registry Mirrors:
  http://65.108.242.32:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 24

 (cluster.py:722, print_all_docker_pieces)
2026-10-27 17:42:39 [ 373 ] DEBUG : Docker networks for project rootteststorages3 are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:733, print_all_docker_pieces)
2026-10-27 17:42:39 [ 373 ] DEBUG : Docker containers for project rootteststorages3 are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:741, print_all_docker_pieces)
2026-10-27 17:42:39 [ 373 ] DEBUG : Docker volumes for project rootteststorages3 are DRIVER    VOLUME NAME
 (cluster.py:749, print_all_docker_pieces)
2026-10-27 17:42:39 [ 373 ] DEBUG : Command:docker container list --all --filter name='^/rootteststorages3_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check)
2026-10-27 17:42:39 [ 373 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup)
2026-10-27 17:42:39 [ 373 ] DEBUG : No running containers for project: rootteststorages3 (cluster.py:782, cleanup)
2026-10-27 17:42:39 [ 373 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup)
2026-10-27 17:42:39 [ 373 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup)
2026-10-27 17:42:39 [ 373 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check)
2026-10-27 17:42:39 [ 373 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check)
2026-10-27 17:42:39 [ 373 ] DEBUG : Images pruned (cluster.py:807, cleanup)
2026-10-27 17:42:39 [ 373 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup)
2026-10-27 17:42:39 [ 373 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check)
2026-10-27 17:42:39 [ 373 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup directory for instance: restricted_dummy (cluster.py:2657, start)
2026-10-27 17:42:39 [ 373 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_s3/configs/config_for_test_remote_host_filter.xml'] to /ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/configs/config.d (cluster.py:4383, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/database (cluster.py:4400, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/logs (cluster.py:4411, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup directory for instance: dummy (cluster.py:2657, start)
2026-10-27 17:42:39 [ 373 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_s3/configs/defaultS3.xml', '/ClickHouse/tests/integration/test_storage_s3/configs/named_collections.xml', '/ClickHouse/tests/integration/test_storage_s3/configs/schema_cache.xml'] to /ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/configs/config.d (cluster.py:4383, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/database (cluster.py:4400, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/logs (cluster.py:4411, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup directory for instance: dummy_without_named_collections (cluster.py:2657, start)
2026-10-27 17:42:39 [ 373 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_s3/configs/defaultS3.xml', '/ClickHouse/tests/integration/test_storage_s3/configs/named_collections.xml', '/ClickHouse/tests/integration/test_storage_s3/configs/schema_cache.xml'] to /ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/configs/config.d (cluster.py:4383, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/database (cluster.py:4400, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/logs (cluster.py:4411, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup directory for instance: s3_max_redirects (cluster.py:2657, start)
2026-10-27 17:42:39 [ 373 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_s3/configs/defaultS3.xml'] to /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/configs/config.d (cluster.py:4383, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/database (cluster.py:4400, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/logs (cluster.py:4411, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup directory for instance: s3_non_default (cluster.py:2657, start)
2026-10-27 17:42:39 [ 373 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_non_default/configs/config.d (cluster.py:4383, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_non_default/database (cluster.py:4400, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_non_default/logs (cluster.py:4411, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup directory for instance: s3_with_environment_credentials (cluster.py:2657, start)
2026-10-27 17:42:39 [ 373 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_s3/configs/use_environment_credentials.xml'] to /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_with_environment_credentials/configs/config.d (cluster.py:4383, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_with_environment_credentials/database (cluster.py:4400, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_with_environment_credentials/logs (cluster.py:4411, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir)
2026-10-27 17:42:39 [ 373 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_storage_s3/_instances_0/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_storage_s3/_instances_0/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_storage_s3/_instances_0/minio/certs/public.crt', 'AWS_ACCESS_KEY_ID': 'minio', 'AWS_SECRET_ACCESS_KEY': 'minio123'} stored in /ClickHouse/tests/integration/test_storage_s3/_instances_0/.env (cluster.py:78, _create_env_file)
2026-10-27 17:42:39 [ 373 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file)
2026-10-27 17:42:39 [ 373 ] DEBUG : No config file found (config.py:28, find_config_file)
2026-10-27 17:42:39 [ 373 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file)
2026-10-27 17:42:39 [ 373 ] DEBUG : No config file found (config.py:28, find_config_file)
2026-10-27 17:42:39 [ 373 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 825 (connectionpool.py:546, _make_request)
2026-10-27 17:42:39 [ 373 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', '--project-name', 'rootteststorages3', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_non_default/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_with_environment_credentials/docker-compose.yml', 'pull'] (cluster.py:105, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling proxy2                          ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling proxy1                          ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling minio1                          ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_non_default                  ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_with_environment_credentials ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling resolver                        ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_max_redirects                ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling restricted_dummy                ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling dummy_without_named_collections ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling dummy                           ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_max_redirects                ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling resolver                        ... pulling from altinityinfra/python... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling resolver                        ... digest: sha256:5a7a0ebe149009e702... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling resolver                        ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling resolver                        ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling proxy2                          ... pulling from altinityinfra/s3-proxy (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling proxy1                          ... pulling from altinityinfra/s3-proxy (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_non_default                  ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_max_redirects                ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_max_redirects                ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling proxy2                          ... digest: sha256:1a92e43060168b5557... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling proxy2                          ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling proxy1                          ... digest: sha256:1a92e43060168b5557... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling proxy1                          ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_max_redirects                ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling proxy2                          ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling proxy1                          ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_non_default                  ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_non_default                  ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_non_default                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling minio1                          ... pulling from clickhouse/minio-minio (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling dummy_without_named_collections ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling restricted_dummy                ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling minio1                          ... digest: sha256:9eea83a4e1425067e7... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling minio1                          ... status: image is up to date for c... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling minio1                          ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_with_environment_credentials ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling dummy                           ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling restricted_dummy                ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling restricted_dummy                ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling dummy_without_named_collections ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling dummy_without_named_collections ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling restricted_dummy                ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling dummy_without_named_collections ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_with_environment_credentials ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_with_environment_credentials ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling s3_with_environment_credentials ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling dummy                           ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling dummy                           ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] DEBUG : Stderr:Pulling dummy                           ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:41 [ 373 ] INFO : Trying to create Minio instance by command docker-compose --env-file /ClickHouse/tests/integration/test_storage_s3/_instances_0/.env --project-name rootteststorages3 --file /compose/docker_compose_minio.yml --verbose up -d (cluster.py:2938, start)
2026-10-27 17:42:41 [ 373 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', '--project-name', 'rootteststorages3', '--file', '/compose/docker_compose_minio.yml', '--verbose', 'up', '-d'] (cluster.py:105, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.config.config.find: Using configuration files: /compose/docker_compose_minio.yml (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.docker_client.get_client: docker-compose version 1.29.2, build unknown (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:docker-py version: <module 'docker.version' from '/usr/local/lib/python3.10/dist-packages/docker/version.py'> (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:CPython version: 3.10.12 (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:OpenSSL version: OpenSSL 3.0.2 15 Mar 2022 (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '23.0.6', 'Details': {'ApiVersion': '1.42', 'Arch': 'amd64', 'BuildTime': '2023-05-05T21:18:13.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9dbdbd4', 'GoVersion': 'go1.19.9', 'KernelVersion': '5.15.0-198-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': 'v2.4.2', 'Details': {'GitCommit': 'b7a25261e029d68978bbf290df0a62fc819b3c51'}}, {'Name': 'runc', 'Version': '1.5.2', 'Details': {'GitCommit': 'v1.5.2-0-g29dd3dc2'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=23.0.6, ApiVersion=1.42, MinAPIVersion=1.12, GitCommit=9dbdbd4, GoVersion=go1.19.9, Os=linux, Arch=amd64, KernelVersion=5.15.0-198-generic, BuildTime=2023-05-05T21:18:13.000000000+00:00 (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('rootteststorages3_default') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('rootteststorages3_data1-1') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info <- () (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info -> {'Architecture': 'x86_64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'BridgeNfIp6tables': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'BridgeNfIptables': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'CPUSet': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'CPUShares': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'CgroupDriver': 'cgroupfs', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'CgroupVersion': '2', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'ContainerdCommit': {'Expected': 'b7a25261e029d68978bbf290df0a62fc819b3c51', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:                      'ID': 'b7a25261e029d68978bbf290df0a62fc819b3c51'}, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Containers': 6, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('rootteststorages3_default') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.network.ensure: Creating network "rootteststorages3_default" with the default driver (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network <- (name='rootteststorages3_default', driver=None, options=None, ipam=None, internal=False, enable_ipv6=False, labels={'com.docker.compose.project': 'rootteststorages3', 'com.docker.compose.network': 'default', 'com.docker.compose.version': '1.29.2'}, attachable=True, check_duplicate=True) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network -> {'Id': 'd9504ed2c6fd290e64fd542dc88ac6340ac611e65fa4349d8930b10c1f3dccaf', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Warning': ''} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('rootteststorages3_data1-1') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.volume.initialize: Creating volume "rootteststorages3_data1-1" with default driver (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_volume <- ('rootteststorages3_data1-1', None, None, labels={'com.docker.compose.project': 'rootteststorages3', 'com.docker.compose.volume': 'data1-1', 'com.docker.compose.version': '1.29.2'}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_volume -> {'CreatedAt': '2026-10-27T17:42:42Z', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Driver': 'local', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Labels': {'com.docker.compose.project': 'rootteststorages3', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'com.docker.compose.version': '1.29.2', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'com.docker.compose.volume': 'data1-1'}, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Mountpoint': '/var/lib/docker/volumes/rootteststorages3_data1-1/_data', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Name': 'rootteststorages3_data1-1', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Options': None, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Scope': 'local'} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['minio'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['python3'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.project._get_convergence_plans: minio1 has upstream changes (proxy1, proxy2) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.project._get_convergence_plans: resolver has upstream changes (proxy1, proxy2) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: proxy2>, <Service: proxy1>, <Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for <Service: proxy2> (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for <Service: proxy1> (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_proxy1_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststorages3', service='proxy1', number=1)} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststorages3', service='proxy1', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_proxy2_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststorages3', service='proxy2', number=1)} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststorages3', service='proxy2', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 043b65d35e7edc3a9b49ef461e31afcef72883d0727981b2f3586f102bf77b2d (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='rootteststorages3_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Links': [], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'NetworkMode': 'rootteststorages3_default', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/s3-proxy', volumes={}, name='rootteststorages3_proxy2_1', detach=True, ports=['8080', '80', '443'], environment=[], labels={'com.docker.compose.project': 'rootteststorages3', 'com.docker.compose.service': 'proxy2', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/compose', 'com.docker.compose.project.config_files': '/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '043b65d35e7edc3a9b49ef461e31afcef72883d0727981b2f3586f102bf77b2d'}, host_config={'NetworkMode': 'rootteststorages3_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststorages3_default': {'Aliases': ['proxy2'], 'IPAMConfig': {}}}}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 043b65d35e7edc3a9b49ef461e31afcef72883d0727981b2f3586f102bf77b2d (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='rootteststorages3_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Links': [], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'NetworkMode': 'rootteststorages3_default', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/s3-proxy', volumes={}, name='rootteststorages3_proxy1_1', detach=True, ports=['8080', '80', '443'], environment=[], labels={'com.docker.compose.project': 'rootteststorages3', 'com.docker.compose.service': 'proxy1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/compose', 'com.docker.compose.project.config_files': '/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '043b65d35e7edc3a9b49ef461e31afcef72883d0727981b2f3586f102bf77b2d'}, host_config={'NetworkMode': 'rootteststorages3_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststorages3_default': {'Aliases': ['proxy1'], 'IPAMConfig': {}}}}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '484cde736474cdcbcb9940521bf3ce449a98114a41b54f61bebe17afa4ee92ae', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Warnings': ["The requested image's platform (linux/arm64/v8) does not match " (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:              'the detected host platform (linux/amd64/v4) and no specific ' (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:              'platform was requested']} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('484cde736474cdcbcb9940521bf3ce449a98114a41b54f61bebe17afa4ee92ae') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Args': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:                    'NGINX_VERSION=1.31.8', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('484cde736474cdcbcb9940521bf3ce449a98114a41b54f61bebe17afa4ee92ae', 'rootteststorages3_default') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('484cde736474cdcbcb9940521bf3ce449a98114a41b54f61bebe17afa4ee92ae', 'rootteststorages3_default', aliases=['proxy2', '484cde736474'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('484cde736474cdcbcb9940521bf3ce449a98114a41b54f61bebe17afa4ee92ae') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '79d44758ae16f0cbd2f48b29b5fb5e028ecb88e45da5b5ec9a6d02e29ed1c214', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Warnings': ["The requested image's platform (linux/arm64/v8) does not match " (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:              'the detected host platform (linux/amd64/v4) and no specific ' (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:              'platform was requested']} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('79d44758ae16f0cbd2f48b29b5fb5e028ecb88e45da5b5ec9a6d02e29ed1c214') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Args': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:                    'NGINX_VERSION=1.31.8', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('79d44758ae16f0cbd2f48b29b5fb5e028ecb88e45da5b5ec9a6d02e29ed1c214', 'rootteststorages3_default') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('79d44758ae16f0cbd2f48b29b5fb5e028ecb88e45da5b5ec9a6d02e29ed1c214', 'rootteststorages3_default', aliases=['proxy1', '79d44758ae16'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('79d44758ae16f0cbd2f48b29b5fb5e028ecb88e45da5b5ec9a6d02e29ed1c214') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststorages3', service='proxy2', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_proxy2_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: <Service: proxy2> (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststorages3', service='proxy1', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_proxy1_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: <Service: proxy1> (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for <Service: minio1> (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for <Service: resolver> (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_minio1_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststorages3', service='minio1', number=1)} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststorages3', service='minio1', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_resolver_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststorages3', service='resolver', number=1)} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststorages3', service='resolver', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['minio'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['python3'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['python3'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 0c82a1d43e92e5319a9f1b74edfe879ae3a2db3fbc06008b468b9d872886ee08 (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='rootteststorages3_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Links': [], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'NetworkMode': 'rootteststorages3_default', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/python-bottle', tty=True, volumes={}, name='rootteststorages3_resolver_1', detach=True, ports=['8080'], environment=[], labels={'com.docker.compose.project': 'rootteststorages3', 'com.docker.compose.service': 'resolver', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/compose', 'com.docker.compose.project.config_files': '/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '0c82a1d43e92e5319a9f1b74edfe879ae3a2db3fbc06008b468b9d872886ee08'}, host_config={'NetworkMode': 'rootteststorages3_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststorages3_default': {'Aliases': ['resolver'], 'IPAMConfig': {}}}}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['minio'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 7e82fc5e5a8a1ccdcb1e9d51335e794e3948ae049f8463fabbaf691fd8a91908 (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=['rootteststorages3_data1-1:/data1:rw', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/minio/certs:/certs:rw'], volumes_from=[], privileged=False, network_mode='rootteststorages3_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': ['rootteststorages3_data1-1:/data1:rw', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:           '/ClickHouse/tests/integration/test_storage_s3/_instances_0/minio/certs:/certs:rw'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Links': [], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'NetworkMode': 'rootteststorages3_default', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (command='server --console-address 127.0.0.1:19001 --address :9001 --certs-dir /certs /data1-1', environment=['MINIO_ACCESS_KEY=minio', 'MINIO_SECRET_KEY=minio123', 'MINIO_PROMETHEUS_AUTH_TYPE=public'], image='clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z', volumes={'/data1': {}, '/certs': {}}, name='rootteststorages3_minio1_1', detach=True, ports=['9001'], labels={'com.docker.compose.project': 'rootteststorages3', 'com.docker.compose.service': 'minio1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/compose', 'com.docker.compose.project.config_files': '/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '7e82fc5e5a8a1ccdcb1e9d51335e794e3948ae049f8463fabbaf691fd8a91908'}, host_config={'NetworkMode': 'rootteststorages3_default', 'VolumesFrom': [], 'Binds': ['rootteststorages3_data1-1:/data1:rw', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/minio/certs:/certs:rw'], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststorages3_default': {'Aliases': ['minio1'], 'IPAMConfig': {}}}}) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '4cc1c4714de84357ee4b893e9b9e6f54ce5ef640918345333becfc5a8c375082', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('4cc1c4714de84357ee4b893e9b9e6f54ce5ef640918345333becfc5a8c375082') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Args': ['server', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:          '--console-address', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:          '127.0.0.1:19001', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:          '--address', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:          ':9001', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:          '--certs-dir', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:          '/certs', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:          '/data1-1'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('4cc1c4714de84357ee4b893e9b9e6f54ce5ef640918345333becfc5a8c375082', 'rootteststorages3_default') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '9af3be0c16be63bf8fcdcdd2781e37a03d73a74a540a6a637c313a14f175661f', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Warnings': ["The requested image's platform (linux/arm64/v8) does not match " (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:              'the detected host platform (linux/amd64/v4) and no specific ' (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:              'platform was requested']} (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('9af3be0c16be63bf8fcdcdd2781e37a03d73a74a540a6a637c313a14f175661f') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Args': [], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Cmd': ['python3'], (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:                    'LANG=C.UTF-8', (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('9af3be0c16be63bf8fcdcdd2781e37a03d73a74a540a6a637c313a14f175661f', 'rootteststorages3_default') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('4cc1c4714de84357ee4b893e9b9e6f54ce5ef640918345333becfc5a8c375082', 'rootteststorages3_default', aliases=['4cc1c4714de8', 'minio1'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('9af3be0c16be63bf8fcdcdd2781e37a03d73a74a540a6a637c313a14f175661f', 'rootteststorages3_default', aliases=['9af3be0c16be', 'resolver'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('4cc1c4714de84357ee4b893e9b9e6f54ce5ef640918345333becfc5a8c375082') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('9af3be0c16be63bf8fcdcdd2781e37a03d73a74a540a6a637c313a14f175661f') (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststorages3', service='minio1', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_minio1_1   ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: <Service: minio1> (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststorages3', service='resolver', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_resolver_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: <Service: resolver> (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:42:44 [ 373 ] INFO : Trying to connect to Minio... (cluster.py:2944, start)
2026-10-27 17:42:44 [ 373 ] DEBUG : get_instance_ip instance_name=minio1 (cluster.py:1971, get_instance_ip)
2026-10-27 17:42:44 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_minio1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:44 [ 373 ] DEBUG : get_instance_ip instance_name=proxy1 (cluster.py:1971, get_instance_ip)
2026-10-27 17:42:44 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_proxy1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:44 [ 373 ] DEBUG : Starting new HTTP connection (1): 172.16.9.4:9001 (connectionpool.py:244, _new_conn)
2026-10-27 17:42:44 [ 373 ] DEBUG : http://172.16.9.4:9001 "GET / HTTP/1.1" 200 0 (connectionpool.py:546, _make_request)
2026-10-27 17:42:44 [ 373 ] DEBUG : Connected to Minio. (cluster.py:2501, wait_minio_to_start)
2026-10-27 17:42:44 [ 373 ] DEBUG : http://172.16.9.4:9001 "GET /root?location= HTTP/1.1" 404 0 (connectionpool.py:546, _make_request)
2026-10-27 17:42:44 [ 373 ] DEBUG : http://172.16.9.4:9001 "PUT /root HTTP/1.1" 200 0 (connectionpool.py:546, _make_request)
2026-10-27 17:42:44 [ 373 ] DEBUG : S3 bucket 'root' created (cluster.py:2516, wait_minio_to_start)
2026-10-27 17:42:44 [ 373 ] DEBUG : http://172.16.9.4:9001 "GET /root2?location= HTTP/1.1" 404 0 (connectionpool.py:546, _make_request)
2026-10-27 17:42:44 [ 373 ] DEBUG : http://172.16.9.4:9001 "PUT /root2 HTTP/1.1" 200 0 (connectionpool.py:546, _make_request)
2026-10-27 17:42:44 [ 373 ] DEBUG : S3 bucket 'root2' created (cluster.py:2516, wait_minio_to_start)
2026-10-27 17:42:44 [ 373 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_storage_s3/_instances_0/.env --project-name rootteststorages3 --file /ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/docker-compose.yml --file /compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_non_default/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_with_environment_credentials/docker-compose.yml up -d --no-recreate') (cluster.py:2980, start)
2026-10-27 17:42:44 [ 373 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', '--project-name', 'rootteststorages3', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_non_default/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_with_environment_credentials/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:105, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_s3_non_default_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_dummy_1          ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_s3_with_environment_credentials_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_restricted_dummy_1                ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_dummy_without_named_collections_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_s3_max_redirects_1                ...  (cluster.py:115, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_s3_with_environment_credentials_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_dummy_without_named_collections_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_s3_max_redirects_1                ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_dummy_1                           ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_restricted_dummy_1                ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : Stderr:Creating rootteststorages3_s3_non_default_1                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:42:47 [ 373 ] DEBUG : ClickHouse instance created (cluster.py:2988, start)
2026-10-27 17:42:47 [ 373 ] DEBUG : get_instance_ip instance_name=restricted_dummy (cluster.py:1971, get_instance_ip)
2026-10-27 17:42:47 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_restricted_dummy_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:47 [ 373 ] DEBUG : Waiting for ClickHouse start in restricted_dummy, ip: 172.16.9.7... (cluster.py:2995, start)
2026-10-27 17:42:47 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_restricted_dummy_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:47 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:47 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:47 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:47 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:47 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:47 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:47 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:47 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:48 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:48 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:48 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:48 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:48 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:48 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:48 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:48 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:48 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:48 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:49 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:49 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:49 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:49 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:49 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:49 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:49 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:49 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:49 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/6370a24a7d5091fe6810bd65684b59e985b2042dc8e283bd816a86c1c6608080/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : ClickHouse restricted_dummy started (cluster.py:2999, start)
2026-10-27 17:42:50 [ 373 ] DEBUG : get_instance_ip instance_name=dummy (cluster.py:1971, get_instance_ip)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_dummy_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : Waiting for ClickHouse start in dummy, ip: 172.16.9.9... (cluster.py:2995, start)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_dummy_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/1fd8fb38228f7c65816a34458313dde719218f11469fbfe9a3419dc9a7e0500f/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : ClickHouse dummy started (cluster.py:2999, start)
2026-10-27 17:42:50 [ 373 ] DEBUG : get_instance_ip instance_name=dummy_without_named_collections (cluster.py:1971, get_instance_ip)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_dummy_without_named_collections_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : Waiting for ClickHouse start in dummy_without_named_collections, ip: 172.16.9.8... (cluster.py:2995, start)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_dummy_without_named_collections_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/79679727404b0c78c36825be586b41ed9f0dc5824cba5a603a0811eccc3d69fd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : ClickHouse dummy_without_named_collections started (cluster.py:2999, start)
2026-10-27 17:42:50 [ 373 ] DEBUG : get_instance_ip instance_name=s3_max_redirects (cluster.py:1971, get_instance_ip)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_s3_max_redirects_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : Waiting for ClickHouse start in s3_max_redirects, ip: 172.16.9.10... (cluster.py:2995, start)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_s3_max_redirects_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/7fceb3caf531be40530cd1988fec996ea950ef1c3c3c2728ce9f1b2e7cce4817/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : ClickHouse s3_max_redirects started (cluster.py:2999, start)
2026-10-27 17:42:50 [ 373 ] DEBUG : get_instance_ip instance_name=s3_non_default (cluster.py:1971, get_instance_ip)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_s3_non_default_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : Waiting for ClickHouse start in s3_non_default, ip: 172.16.9.11... (cluster.py:2995, start)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_s3_non_default_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/3959b12cbc9a07ac36f2b8f9da1512de078853e432d71c6ccd143f15b450df49/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : ClickHouse s3_non_default started (cluster.py:2999, start)
2026-10-27 17:42:50 [ 373 ] DEBUG : get_instance_ip instance_name=s3_with_environment_credentials (cluster.py:1971, get_instance_ip)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_s3_with_environment_credentials_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : Waiting for ClickHouse start in s3_with_environment_credentials, ip: 172.16.9.6... (cluster.py:2995, start)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3_s3_with_environment_credentials_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://localhost:None "GET /v1.42/containers/1d438d9a8c96db077c08b29ce1de9a592a9f661398cd54193209a34df725c4f8/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : ClickHouse s3_with_environment_credentials started (cluster.py:2999, start)
2026-10-27 17:42:50 [ 373 ] INFO : Cluster started (test.py:92, started_cluster)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://172.16.9.4:9001 "PUT /root?policy= HTTP/1.1" 204 0 (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://172.16.9.4:9001 "GET /root-with-auth?location= HTTP/1.1" 404 0 (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] DEBUG : http://172.16.9.4:9001 "PUT /root-with-auth HTTP/1.1" 200 0 (connectionpool.py:546, _make_request)
2026-10-27 17:42:50 [ 373 ] INFO : S3 bucket created (test.py:95, started_cluster)
2026-10-27 17:42:50 [ 373 ] INFO : Starting mock servers mock_s3.py,unstable_server.py,echo.py,no_list_objects.py (mock_servers.py:18, start_mock_servers)
2026-10-27 17:42:50 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJvdXRlLCBydW4sIHJlcXVlc3QsIHJlc3BvbnNlCgoKQHJvdXRlKCIvcmVkaXJlY3RlZC88X3BhdGg6cGF0aD4iKQpkZWYgaW5maW5pdGVfcmVkaXJlY3QoX3BhdGgpOgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiTG9jYXRpb24iLCByZXF1ZXN0LnVybCkKICAgIHJlc3BvbnNlLnN0YXR1cyA9IDMwNwogICAgcmV0dXJuICJSZWRpcmVjdGVkIgoKCkByb3V0ZSgiLzxfYnVja2V0Pi88X3BhdGg6cGF0aD4iKQpkZWYgc2VydmVyKF9idWNrZXQsIF9wYXRoKToKICAgIGZvciBuYW1lIGluIHJlcXVlc3QuaGVhZGVyczoKICAgICAgICBpZiBuYW1lID09ICJBdXRob3JpemF0aW9uIiBhbmQgcmVxdWVzdC5oZWFkZXJzW25hbWVdID09ICJCZWFyZXIgVE9LRU4iOgogICAgICAgICAgICByZXN1bHQgPSAiMSwgMiwgMyIKICAgICAgICAgICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICAgICAgICAgIHJlc3BvbnNlLnNldF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgbGVuKHJlc3VsdCkpCiAgICAgICAgICAgIHJldHVybiByZXN1bHQKCiAgICByZXN1bHQgPSAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48RXJyb3I+PENvZGU+Rm9yYmlkZGVuRXJyb3I8L0NvZGU+PE1lc3NhZ2U+Rm9yYmlkZGVuIEVycm9yPC9NZXNzYWdlPjxSZXF1ZXN0SWQ+dHhmYmQ1NjZkMDMwNDI0NzQ4ODgxOTMtMDA2MDhkNzUzNzwvUmVxdWVzdElkPjwvRXJyb3I+JwogICAgcmVzcG9uc2Uuc3RhdHVzID0gNDAzCiAgICByZXNwb25zZS5jb250ZW50X3R5cGUgPSAidGV4dC94bWwiCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIGxlbihyZXN1bHQpKQogICAgcmV0dXJuIHJlc3VsdAoKCkByb3V0ZSgiLyIpCmRlZiBwaW5nKCk6CiAgICByZXNwb25zZS5jb250ZW50X3R5cGUgPSAidGV4dC9wbGFpbiIKICAgIHJlc3BvbnNlLnNldF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgMikKICAgIHJldHVybiAiT0siCgoKcnVuKGhvc3Q9IjAuMC4wLjAiLCBwb3J0PWludChzeXMuYXJndlsxXSkpCg== | base64 --decode > mock_s3.py'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:50 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'bash', '-c', 'echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJvdXRlLCBydW4sIHJlcXVlc3QsIHJlc3BvbnNlCgoKQHJvdXRlKCIvcmVkaXJlY3RlZC88X3BhdGg6cGF0aD4iKQpkZWYgaW5maW5pdGVfcmVkaXJlY3QoX3BhdGgpOgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiTG9jYXRpb24iLCByZXF1ZXN0LnVybCkKICAgIHJlc3BvbnNlLnN0YXR1cyA9IDMwNwogICAgcmV0dXJuICJSZWRpcmVjdGVkIgoKCkByb3V0ZSgiLzxfYnVja2V0Pi88X3BhdGg6cGF0aD4iKQpkZWYgc2VydmVyKF9idWNrZXQsIF9wYXRoKToKICAgIGZvciBuYW1lIGluIHJlcXVlc3QuaGVhZGVyczoKICAgICAgICBpZiBuYW1lID09ICJBdXRob3JpemF0aW9uIiBhbmQgcmVxdWVzdC5oZWFkZXJzW25hbWVdID09ICJCZWFyZXIgVE9LRU4iOgogICAgICAgICAgICByZXN1bHQgPSAiMSwgMiwgMyIKICAgICAgICAgICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICAgICAgICAgIHJlc3BvbnNlLnNldF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgbGVuKHJlc3VsdCkpCiAgICAgICAgICAgIHJldHVybiByZXN1bHQKCiAgICByZXN1bHQgPSAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48RXJyb3I+PENvZGU+Rm9yYmlkZGVuRXJyb3I8L0NvZGU+PE1lc3NhZ2U+Rm9yYmlkZGVuIEVycm9yPC9NZXNzYWdlPjxSZXF1ZXN0SWQ+dHhmYmQ1NjZkMDMwNDI0NzQ4ODgxOTMtMDA2MDhkNzUzNzwvUmVxdWVzdElkPjwvRXJyb3I+JwogICAgcmVzcG9uc2Uuc3RhdHVzID0gNDAzCiAgICByZXNwb25zZS5jb250ZW50X3R5cGUgPSAidGV4dC94bWwiCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIGxlbihyZXN1bHQpKQogICAgcmV0dXJuIHJlc3VsdAoKCkByb3V0ZSgiLyIpCmRlZiBwaW5nKCk6CiAgICByZXNwb25zZS5jb250ZW50X3R5cGUgPSAidGV4dC9wbGFpbiIKICAgIHJlc3BvbnNlLnNldF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgMikKICAgIHJldHVybiAiT0siCgoKcnVuKGhvc3Q9IjAuMC4wLjAiLCBwb3J0PWludChzeXMuYXJndlsxXSkpCg== | base64 --decode > mock_s3.py'] (cluster.py:105, run_and_check)
2026-10-27 17:42:51 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:True nothrow:False cmd: ['python', 'mock_s3.py', '8080'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:51 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo aW1wb3J0IGh0dHAuc2VydmVyCmltcG9ydCByYW5kb20KaW1wb3J0IHJlCmltcG9ydCBzb2NrZXQKaW1wb3J0IHN0cnVjdAppbXBvcnQgc3lzCgoKZGVmIGdlbl9uX2RpZ2l0X251bWJlcihuKToKICAgIGFzc2VydCAwIDwgbiA8IDE5CiAgICByZXR1cm4gcmFuZG9tLnJhbmRpbnQoMTAgKiogKG4gLSAxKSwgMTAqKm4gLSAxKQoKCnN1bV9pbl80X2NvbHVtbiA9IDAKCgpkZWYgZ2VuX2xpbmUoKToKICAgIGdsb2JhbCBzdW1faW5fNF9jb2x1bW4KICAgIGNvbHVtbnMgPSA0CgogICAgcm93ID0gW10KCiAgICBkZWYgYWRkX251bWJlcigpOgogICAgICAgIGRpZ2l0cyA9IHJhbmRvbS5yYW5kaW50KDEsIDE4KQogICAgICAgIHJvdy5hcHBlbmQoZ2VuX25fZGlnaXRfbnVtYmVyKGRpZ2l0cykpCgogICAgZm9yIGkgaW4gcmFuZ2UoY29sdW1ucyAvLyAyKToKICAgICAgICBhZGRfbnVtYmVyKCkKICAgIHJvdy5hcHBlbmQoMSkKICAgIGZvciBpIGluIHJhbmdlKGNvbHVtbnMgLSAxIC0gY29sdW1ucyAvLyAyKToKICAgICAgICBhZGRfbnVtYmVyKCkKICAgIHN1bV9pbl80X2NvbHVtbiArPSByb3dbLTFdCgogICAgbGluZSA9ICIsIi5qb2luKG1hcChzdHIsIHJvdykpICsgIlxuIgogICAgcmV0dXJuIGxpbmUuZW5jb2RlKCkKCgpyYW5kb20uc2VlZCgiVW5zdGFibGUgc2VydmVyLzEuMCIpCgojIEdlbmVyYXRpbmcgc29tZSAicmFuZG9tIiBkYXRhIGFuZCBhcHBlbmQgYSBsaW5lIHdoaWNoIGNvbnRhaW5zIHN1bSBvZiBudW1iZXJzIGluIGNvbHVtbiA0LgpsaW5lcyA9ICgKICAgIGIiIi5qb2luKChnZW5fbGluZSgpIGZvciBfIGluIHJhbmdlKDUwMDAwMCkpKQogICAgKyBmIjAsMCwwLHstc3VtX2luXzRfY29sdW1ufVxuIi5lbmNvZGUoKQopCgoKY2xhc3MgUmVxdWVzdEhhbmRsZXIoaHR0cC5zZXJ2ZXIuQmFzZUhUVFBSZXF1ZXN0SGFuZGxlcik6CiAgICBkZWYgZG9fSEVBRChzZWxmKToKICAgICAgICBpZiBzZWxmLnBhdGggPT0gIi9yb290L3Rlc3QuY3N2IjoKICAgICAgICAgICAgc2VsZi5mcm9tX2J5dGVzID0gMAogICAgICAgICAgICBzZWxmLmVuZF9ieXRlcyA9IGxlbihsaW5lcykKICAgICAgICAgICAgc2VsZi5zaXplID0gc2VsZi5lbmRfYnl0ZXMKICAgICAgICAgICAgc2VsZi5zZW5kX2Jsb2NrX3NpemUgPSAyNTYKICAgICAgICAgICAgc2VsZi5zdG9wX2F0ID0gKAogICAgICAgICAgICAgICAgcmFuZG9tLnJhbmRpbnQoOTAwMDAwLCAxMzAwMDAwKSAvLyBzZWxmLnNlbmRfYmxvY2tfc2l6ZQogICAgICAgICAgICApICAjIEJsb2NrIHNpemUgaXMgMTAyNCoqMi4KCiAgICAgICAgICAgIGlmICJSYW5nZSIgaW4gc2VsZi5oZWFkZXJzOgogICAgICAgICAgICAgICAgY3IgPSBzZWxmLmhlYWRlcnNbIlJhbmdlIl0KICAgICAgICAgICAgICAgIHBhcnRzID0gcmUuc3BsaXQoIlsgLS89XSsiLCBjcikKICAgICAgICAgICAgICAgIGFzc2VydCBwYXJ0c1swXSA9PSAiYnl0ZXMiCiAgICAgICAgICAgICAgICBzZWxmLmZyb21fYnl0ZXMgPSBpbnQocGFydHNbMV0pCiAgICAgICAgICAgICAgICBpZiBwYXJ0c1syXToKICAgICAgICAgICAgICAgICAgICBzZWxmLmVuZF9ieXRlcyA9IGludChwYXJ0c1syXSkgKyAxCiAgICAgICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjA2KQogICAgICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigKICAgICAgICAgICAgICAgICAgICAiQ29udGVudC1SYW5nZSIsCiAgICAgICAgICAgICAgICAgICAgZiJieXRlcyB7c2VsZi5mcm9tX2J5dGVzfS17c2VsZi5lbmRfYnl0ZXMtMX0ve3NlbGYuc2l6ZX0iLAogICAgICAgICAgICAgICAgKQogICAgICAgICAgICBlbHNlOgogICAgICAgICAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDIwMCkKCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkFjY2VwdC1SYW5nZXMiLCAiYnl0ZXMiKQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAidGV4dC9wbGFpbiIpCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgZiJ7c2VsZi5lbmRfYnl0ZXMtc2VsZi5mcm9tX2J5dGVzfSIpCiAgICAgICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQoKICAgICAgICBlbGlmIHNlbGYucGF0aCA9PSAiLyI6CiAgICAgICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3BsYWluIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgIGVsc2U6CiAgICAgICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSg0MDQpCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3BsYWluIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgZGVmIGRvX0dFVChzZWxmKToKICAgICAgICBzZWxmLmRvX0hFQUQoKQogICAgICAgIGlmIHNlbGYucGF0aCA9PSAiL3Jvb3QvdGVzdC5jc3YiOgogICAgICAgICAgICBmb3IgYywgaSBpbiBlbnVtZXJhdGUoCiAgICAgICAgICAgICAgICByYW5nZShzZWxmLmZyb21fYnl0ZXMsIHNlbGYuZW5kX2J5dGVzLCBzZWxmLnNlbmRfYmxvY2tfc2l6ZSkKICAgICAgICAgICAgKToKICAgICAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoCiAgICAgICAgICAgICAgICAgICAgbGluZXNbaSA6IG1pbihpICsgc2VsZi5zZW5kX2Jsb2NrX3NpemUsIHNlbGYuZW5kX2J5dGVzKV0KICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgIGlmIChjICsgMSkgJSBzZWxmLnN0b3BfYXQgPT0gMDoKICAgICAgICAgICAgICAgICAgICAjIHNlbGYud2ZpbGUuX3NvY2suc2V0c29ja29wdChzb2NrZXQuU09MX1NPQ0tFVCwgc29ja2V0LlNPX0xJTkdFUiwgc3RydWN0LnBhY2soImlpIiwgMCwgMCkpCiAgICAgICAgICAgICAgICAgICAgIyBzZWxmLndmaWxlLl9zb2NrLnNodXRkb3duKHNvY2tldC5TSFVUX1JEV1IpCiAgICAgICAgICAgICAgICAgICAgIyBzZWxmLndmaWxlLl9zb2NrLmNsb3NlKCkKICAgICAgICAgICAgICAgICAgICBwcmludCgiRHJvcHBpbmcgY29ubmVjdGlvbiIpCiAgICAgICAgICAgICAgICAgICAgYnJlYWsKCiAgICAgICAgZWxpZiBzZWxmLnBhdGggPT0gIi8iOgogICAgICAgICAgICBzZWxmLndmaWxlLndyaXRlKGIiT0siKQoKCmh0dHBkID0gaHR0cC5zZXJ2ZXIuSFRUUFNlcnZlcigoIjAuMC4wLjAiLCBpbnQoc3lzLmFyZ3ZbMV0pKSwgUmVxdWVzdEhhbmRsZXIpCmh0dHBkLnNlcnZlX2ZvcmV2ZXIoKQo= | base64 --decode > unstable_server.py'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:51 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'bash', '-c', 'echo aW1wb3J0IGh0dHAuc2VydmVyCmltcG9ydCByYW5kb20KaW1wb3J0IHJlCmltcG9ydCBzb2NrZXQKaW1wb3J0IHN0cnVjdAppbXBvcnQgc3lzCgoKZGVmIGdlbl9uX2RpZ2l0X251bWJlcihuKToKICAgIGFzc2VydCAwIDwgbiA8IDE5CiAgICByZXR1cm4gcmFuZG9tLnJhbmRpbnQoMTAgKiogKG4gLSAxKSwgMTAqKm4gLSAxKQoKCnN1bV9pbl80X2NvbHVtbiA9IDAKCgpkZWYgZ2VuX2xpbmUoKToKICAgIGdsb2JhbCBzdW1faW5fNF9jb2x1bW4KICAgIGNvbHVtbnMgPSA0CgogICAgcm93ID0gW10KCiAgICBkZWYgYWRkX251bWJlcigpOgogICAgICAgIGRpZ2l0cyA9IHJhbmRvbS5yYW5kaW50KDEsIDE4KQogICAgICAgIHJvdy5hcHBlbmQoZ2VuX25fZGlnaXRfbnVtYmVyKGRpZ2l0cykpCgogICAgZm9yIGkgaW4gcmFuZ2UoY29sdW1ucyAvLyAyKToKICAgICAgICBhZGRfbnVtYmVyKCkKICAgIHJvdy5hcHBlbmQoMSkKICAgIGZvciBpIGluIHJhbmdlKGNvbHVtbnMgLSAxIC0gY29sdW1ucyAvLyAyKToKICAgICAgICBhZGRfbnVtYmVyKCkKICAgIHN1bV9pbl80X2NvbHVtbiArPSByb3dbLTFdCgogICAgbGluZSA9ICIsIi5qb2luKG1hcChzdHIsIHJvdykpICsgIlxuIgogICAgcmV0dXJuIGxpbmUuZW5jb2RlKCkKCgpyYW5kb20uc2VlZCgiVW5zdGFibGUgc2VydmVyLzEuMCIpCgojIEdlbmVyYXRpbmcgc29tZSAicmFuZG9tIiBkYXRhIGFuZCBhcHBlbmQgYSBsaW5lIHdoaWNoIGNvbnRhaW5zIHN1bSBvZiBudW1iZXJzIGluIGNvbHVtbiA0LgpsaW5lcyA9ICgKICAgIGIiIi5qb2luKChnZW5fbGluZSgpIGZvciBfIGluIHJhbmdlKDUwMDAwMCkpKQogICAgKyBmIjAsMCwwLHstc3VtX2luXzRfY29sdW1ufVxuIi5lbmNvZGUoKQopCgoKY2xhc3MgUmVxdWVzdEhhbmRsZXIoaHR0cC5zZXJ2ZXIuQmFzZUhUVFBSZXF1ZXN0SGFuZGxlcik6CiAgICBkZWYgZG9fSEVBRChzZWxmKToKICAgICAgICBpZiBzZWxmLnBhdGggPT0gIi9yb290L3Rlc3QuY3N2IjoKICAgICAgICAgICAgc2VsZi5mcm9tX2J5dGVzID0gMAogICAgICAgICAgICBzZWxmLmVuZF9ieXRlcyA9IGxlbihsaW5lcykKICAgICAgICAgICAgc2VsZi5zaXplID0gc2VsZi5lbmRfYnl0ZXMKICAgICAgICAgICAgc2VsZi5zZW5kX2Jsb2NrX3NpemUgPSAyNTYKICAgICAgICAgICAgc2VsZi5zdG9wX2F0ID0gKAogICAgICAgICAgICAgICAgcmFuZG9tLnJhbmRpbnQoOTAwMDAwLCAxMzAwMDAwKSAvLyBzZWxmLnNlbmRfYmxvY2tfc2l6ZQogICAgICAgICAgICApICAjIEJsb2NrIHNpemUgaXMgMTAyNCoqMi4KCiAgICAgICAgICAgIGlmICJSYW5nZSIgaW4gc2VsZi5oZWFkZXJzOgogICAgICAgICAgICAgICAgY3IgPSBzZWxmLmhlYWRlcnNbIlJhbmdlIl0KICAgICAgICAgICAgICAgIHBhcnRzID0gcmUuc3BsaXQoIlsgLS89XSsiLCBjcikKICAgICAgICAgICAgICAgIGFzc2VydCBwYXJ0c1swXSA9PSAiYnl0ZXMiCiAgICAgICAgICAgICAgICBzZWxmLmZyb21fYnl0ZXMgPSBpbnQocGFydHNbMV0pCiAgICAgICAgICAgICAgICBpZiBwYXJ0c1syXToKICAgICAgICAgICAgICAgICAgICBzZWxmLmVuZF9ieXRlcyA9IGludChwYXJ0c1syXSkgKyAxCiAgICAgICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjA2KQogICAgICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigKICAgICAgICAgICAgICAgICAgICAiQ29udGVudC1SYW5nZSIsCiAgICAgICAgICAgICAgICAgICAgZiJieXRlcyB7c2VsZi5mcm9tX2J5dGVzfS17c2VsZi5lbmRfYnl0ZXMtMX0ve3NlbGYuc2l6ZX0iLAogICAgICAgICAgICAgICAgKQogICAgICAgICAgICBlbHNlOgogICAgICAgICAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDIwMCkKCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkFjY2VwdC1SYW5nZXMiLCAiYnl0ZXMiKQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAidGV4dC9wbGFpbiIpCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgZiJ7c2VsZi5lbmRfYnl0ZXMtc2VsZi5mcm9tX2J5dGVzfSIpCiAgICAgICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQoKICAgICAgICBlbGlmIHNlbGYucGF0aCA9PSAiLyI6CiAgICAgICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3BsYWluIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgIGVsc2U6CiAgICAgICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSg0MDQpCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3BsYWluIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgZGVmIGRvX0dFVChzZWxmKToKICAgICAgICBzZWxmLmRvX0hFQUQoKQogICAgICAgIGlmIHNlbGYucGF0aCA9PSAiL3Jvb3QvdGVzdC5jc3YiOgogICAgICAgICAgICBmb3IgYywgaSBpbiBlbnVtZXJhdGUoCiAgICAgICAgICAgICAgICByYW5nZShzZWxmLmZyb21fYnl0ZXMsIHNlbGYuZW5kX2J5dGVzLCBzZWxmLnNlbmRfYmxvY2tfc2l6ZSkKICAgICAgICAgICAgKToKICAgICAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoCiAgICAgICAgICAgICAgICAgICAgbGluZXNbaSA6IG1pbihpICsgc2VsZi5zZW5kX2Jsb2NrX3NpemUsIHNlbGYuZW5kX2J5dGVzKV0KICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgIGlmIChjICsgMSkgJSBzZWxmLnN0b3BfYXQgPT0gMDoKICAgICAgICAgICAgICAgICAgICAjIHNlbGYud2ZpbGUuX3NvY2suc2V0c29ja29wdChzb2NrZXQuU09MX1NPQ0tFVCwgc29ja2V0LlNPX0xJTkdFUiwgc3RydWN0LnBhY2soImlpIiwgMCwgMCkpCiAgICAgICAgICAgICAgICAgICAgIyBzZWxmLndmaWxlLl9zb2NrLnNodXRkb3duKHNvY2tldC5TSFVUX1JEV1IpCiAgICAgICAgICAgICAgICAgICAgIyBzZWxmLndmaWxlLl9zb2NrLmNsb3NlKCkKICAgICAgICAgICAgICAgICAgICBwcmludCgiRHJvcHBpbmcgY29ubmVjdGlvbiIpCiAgICAgICAgICAgICAgICAgICAgYnJlYWsKCiAgICAgICAgZWxpZiBzZWxmLnBhdGggPT0gIi8iOgogICAgICAgICAgICBzZWxmLndmaWxlLndyaXRlKGIiT0siKQoKCmh0dHBkID0gaHR0cC5zZXJ2ZXIuSFRUUFNlcnZlcigoIjAuMC4wLjAiLCBpbnQoc3lzLmFyZ3ZbMV0pKSwgUmVxdWVzdEhhbmRsZXIpCmh0dHBkLnNlcnZlX2ZvcmV2ZXIoKQo= | base64 --decode > unstable_server.py'] (cluster.py:105, run_and_check)
2026-10-27 17:42:51 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:True nothrow:False cmd: ['python', 'unstable_server.py', '8081'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:51 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo aW1wb3J0IGh0dHAuc2VydmVyCmltcG9ydCBzeXMKCgpjbGFzcyBSZXF1ZXN0SGFuZGxlcihodHRwLnNlcnZlci5CYXNlSFRUUFJlcXVlc3RIYW5kbGVyKToKICAgIGRlZiBnZXRfcmVzcG9uc2Uoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9nZXQtbXktcGF0aC8iKToKICAgICAgICAgICAgcmV0dXJuIGIiLyIgKyBzZWxmLnBhdGguc3BsaXQoIi8iLCBtYXhzcGxpdD0yKVsyXS5lbmNvZGUoKQogICAgICAgIGVsaWYgc2VsZi5wYXRoID09ICIvIjoKICAgICAgICAgICAgcmV0dXJuIGIiT0siCgogICAgICAgIHJldHVybiBOb25lCgogICAgZGVmIGRvX0hFQUQoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9nZXQtbXktcGF0aC8iKSBvciBzZWxmLnBhdGggPT0gIi8iOgogICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAidGV4dC9wbGFpbiIpCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgbGVuKHNlbGYuZ2V0X3Jlc3BvbnNlKCkpKQogICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICBlbHNlOgogICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNDA0KQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAidGV4dC9wbGFpbiIpCiAgICAgICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQoKICAgIGRlZiBkb19HRVQoc2VsZik6CiAgICAgICAgc2VsZi5kb19IRUFEKCkKICAgICAgICBzZWxmLndmaWxlLndyaXRlKHNlbGYuZ2V0X3Jlc3BvbnNlKCkpCgoKaHR0cGQgPSBodHRwLnNlcnZlci5IVFRQU2VydmVyKCgiMC4wLjAuMCIsIGludChzeXMuYXJndlsxXSkpLCBSZXF1ZXN0SGFuZGxlcikKaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > echo.py'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:51 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'bash', '-c', 'echo aW1wb3J0IGh0dHAuc2VydmVyCmltcG9ydCBzeXMKCgpjbGFzcyBSZXF1ZXN0SGFuZGxlcihodHRwLnNlcnZlci5CYXNlSFRUUFJlcXVlc3RIYW5kbGVyKToKICAgIGRlZiBnZXRfcmVzcG9uc2Uoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9nZXQtbXktcGF0aC8iKToKICAgICAgICAgICAgcmV0dXJuIGIiLyIgKyBzZWxmLnBhdGguc3BsaXQoIi8iLCBtYXhzcGxpdD0yKVsyXS5lbmNvZGUoKQogICAgICAgIGVsaWYgc2VsZi5wYXRoID09ICIvIjoKICAgICAgICAgICAgcmV0dXJuIGIiT0siCgogICAgICAgIHJldHVybiBOb25lCgogICAgZGVmIGRvX0hFQUQoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9nZXQtbXktcGF0aC8iKSBvciBzZWxmLnBhdGggPT0gIi8iOgogICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAidGV4dC9wbGFpbiIpCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgbGVuKHNlbGYuZ2V0X3Jlc3BvbnNlKCkpKQogICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICBlbHNlOgogICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNDA0KQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAidGV4dC9wbGFpbiIpCiAgICAgICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQoKICAgIGRlZiBkb19HRVQoc2VsZik6CiAgICAgICAgc2VsZi5kb19IRUFEKCkKICAgICAgICBzZWxmLndmaWxlLndyaXRlKHNlbGYuZ2V0X3Jlc3BvbnNlKCkpCgoKaHR0cGQgPSBodHRwLnNlcnZlci5IVFRQU2VydmVyKCgiMC4wLjAuMCIsIGludChzeXMuYXJndlsxXSkpLCBSZXF1ZXN0SGFuZGxlcikKaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > echo.py'] (cluster.py:105, run_and_check)
2026-10-27 17:42:51 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:True nothrow:False cmd: ['python', 'echo.py', '8082'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:51 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo aW1wb3J0IGh0dHAuY2xpZW50CmltcG9ydCBodHRwLnNlcnZlcgppbXBvcnQgcmFuZG9tCmltcG9ydCBzb2NrZXRzZXJ2ZXIKaW1wb3J0IHN5cwppbXBvcnQgdXJsbGliLnBhcnNlCgoKVVBTVFJFQU1fSE9TVCA9ICJtaW5pbzE6OTAwMSIKcmFuZG9tLnNlZWQoIk5vIGxpc3Qgb2JqZWN0cy8xLjAiKQoKbGlzdF9yZXF1ZXN0X2NvdW50ZXIgPSAwCmxpc3RfcmVxdWVzdF9tYXhfbnVtYmVyID0gMTAKCgpkZWYgcmVxdWVzdChjb21tYW5kLCB1cmwsIGhlYWRlcnM9e30sIGRhdGE9Tm9uZSk6CiAgICAiIiJNaW5pLXJlcXVlc3RzLiIiIgoKICAgIGNsYXNzIER1bW15OgogICAgICAgIHBhc3MKCiAgICBwYXJ0cyA9IHVybGxpYi5wYXJzZS51cmxwYXJzZSh1cmwpCiAgICBjID0gaHR0cC5jbGllbnQuSFRUUENvbm5lY3Rpb24ocGFydHMuaG9zdG5hbWUsIHBhcnRzLnBvcnQpCiAgICBjLnJlcXVlc3QoCiAgICAgICAgY29tbWFuZCwKICAgICAgICB1cmxsaWIucGFyc2UudXJsdW5wYXJzZShwYXJ0cy5fcmVwbGFjZShzY2hlbWU9IiIsIG5ldGxvYz0iIikpLAogICAgICAgIGhlYWRlcnM9aGVhZGVycywKICAgICAgICBib2R5PWRhdGEsCiAgICApCiAgICByID0gYy5nZXRyZXNwb25zZSgpCiAgICByZXN1bHQgPSBEdW1teSgpCiAgICByZXN1bHQuc3RhdHVzX2NvZGUgPSByLnN0YXR1cwogICAgcmVzdWx0LmhlYWRlcnMgPSByLmhlYWRlcnMKICAgIHJlc3VsdC5jb250ZW50ID0gci5yZWFkKCkKICAgIHJldHVybiByZXN1bHQKCgpjbGFzcyBSZXF1ZXN0SGFuZGxlcihodHRwLnNlcnZlci5CYXNlSFRUUFJlcXVlc3RIYW5kbGVyKToKICAgIGRlZiBkb19HRVQoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoID09ICIvIjoKICAgICAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDIwMCkKICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQvcGxhaW4iKQogICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICAgICAgc2VsZi53ZmlsZS53cml0ZShiIk9LIikKICAgICAgICAgICAgcmV0dXJuCgogICAgICAgIHF1ZXJ5ID0gdXJsbGliLnBhcnNlLnVybHBhcnNlKHNlbGYucGF0aCkucXVlcnkKICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCgogICAgICAgIGdsb2JhbCBsaXN0X3JlcXVlc3RfY291bnRlcgogICAgICAgIGdsb2JhbCBsaXN0X3JlcXVlc3RfbWF4X251bWJlcgoKICAgICAgICBpZiAibGlzdC10eXBlIiBpbiBwYXJhbXM6CiAgICAgICAgICAgIGlmIGxpc3RfcmVxdWVzdF9jb3VudGVyID4gbGlzdF9yZXF1ZXN0X21heF9udW1iZXI6CiAgICAgICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNTAxKQogICAgICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgImFwcGxpY2F0aW9uL3htbCIpCiAgICAgICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoCiAgICAgICAgICAgICAgICAgICAgYiIiIjw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+CiAgICAgICAgICAgICAgICAgICAgICAgIDxFcnJvcj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxDb2RlPk5vdEltcGxlbWVudGVkPC9Db2RlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPE1lc3NhZ2U+SSBjYW4gbGlzdCBvYmplY3Qgb25seSBvbmNlPC9NZXNzYWdlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPFJlc291cmNlPlJFU09VUkNFPC9SZXNvdXJjZT4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxSZXF1ZXN0SWQ+UkVRVUVTVF9JRDwvUmVxdWVzdElkPgogICAgICAgICAgICAgICAgICAgICAgICA8L0Vycm9yPgogICAgICAgICAgICAgICAgICAgICIiIgogICAgICAgICAgICAgICAgKQogICAgICAgICAgICBlbHNlOgogICAgICAgICAgICAgICAgbGlzdF9yZXF1ZXN0X2NvdW50ZXIgKz0gMQogICAgICAgICAgICAgICAgc2VsZi5kb19IRUFEKCkKICAgICAgICBlbHNlOgogICAgICAgICAgICBzZWxmLmRvX0hFQUQoKQoKICAgIGRlZiBkb19QVVQoc2VsZik6CiAgICAgICAgc2VsZi5kb19IRUFEKCkKCiAgICBkZWYgZG9fREVMRVRFKHNlbGYpOgogICAgICAgIHNlbGYuZG9fSEVBRCgpCgogICAgZGVmIGRvX1BPU1Qoc2VsZik6CiAgICAgICAgZ2xvYmFsIGxpc3RfcmVxdWVzdF9jb3VudGVyCiAgICAgICAgZ2xvYmFsIGxpc3RfcmVxdWVzdF9tYXhfbnVtYmVyCgogICAgICAgIGlmIHNlbGYucGF0aC5zdGFydHN3aXRoKCIvcmVzZXRfY291bnRlcnMiKToKICAgICAgICAgICAgcXVlcnkgPSB1cmxsaWIucGFyc2UudXJscGFyc2Uoc2VsZi5wYXRoKS5xdWVyeQogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCgogICAgICAgICAgICBsaXN0X3JlcXVlc3RfY291bnRlciA9IDAKICAgICAgICAgICAgaWYgIm1heCIgaW4gcGFyYW1zOgogICAgICAgICAgICAgICAgbGlzdF9yZXF1ZXN0X21heF9udW1iZXIgPSBpbnQocGFyYW1zWyJtYXgiXVswXSkKCiAgICAgICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3BsYWluIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCiAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYiJPSyIpCiAgICAgICAgICAgIHJldHVybgoKICAgICAgICBzZWxmLmRvX0hFQUQoKQoKICAgIGRlZiBkb19IRUFEKHNlbGYpOgogICAgICAgIGNvbnRlbnRfbGVuZ3RoID0gc2VsZi5oZWFkZXJzLmdldCgiQ29udGVudC1MZW5ndGgiKQogICAgICAgIGRhdGEgPSBzZWxmLnJmaWxlLnJlYWQoaW50KGNvbnRlbnRfbGVuZ3RoKSkgaWYgY29udGVudF9sZW5ndGggZWxzZSBOb25lCiAgICAgICAgciA9IHJlcXVlc3QoCiAgICAgICAgICAgIHNlbGYuY29tbWFuZCwKICAgICAgICAgICAgZiJodHRwOi8ve1VQU1RSRUFNX0hPU1R9e3NlbGYucGF0aH0iLAogICAgICAgICAgICBoZWFkZXJzPXNlbGYuaGVhZGVycywKICAgICAgICAgICAgZGF0YT1kYXRhLAogICAgICAgICkKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2Uoci5zdGF0dXNfY29kZSkKICAgICAgICBmb3IgaywgdiBpbiByLmhlYWRlcnMuaXRlbXMoKToKICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcihrLCB2KQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoci5jb250ZW50KQogICAgICAgIHNlbGYud2ZpbGUuY2xvc2UoKQoKCmNsYXNzIFRocmVhZGVkSFRUUFNlcnZlcihzb2NrZXRzZXJ2ZXIuVGhyZWFkaW5nTWl4SW4sIGh0dHAuc2VydmVyLkhUVFBTZXJ2ZXIpOgogICAgIiIiSGFuZGxlIHJlcXVlc3RzIGluIGEgc2VwYXJhdGUgdGhyZWFkLiIiIgoKCmh0dHBkID0gVGhyZWFkZWRIVFRQU2VydmVyKCgiMC4wLjAuMCIsIGludChzeXMuYXJndlsxXSkpLCBSZXF1ZXN0SGFuZGxlcikKaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > no_list_objects.py'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:51 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'bash', '-c', 'echo aW1wb3J0IGh0dHAuY2xpZW50CmltcG9ydCBodHRwLnNlcnZlcgppbXBvcnQgcmFuZG9tCmltcG9ydCBzb2NrZXRzZXJ2ZXIKaW1wb3J0IHN5cwppbXBvcnQgdXJsbGliLnBhcnNlCgoKVVBTVFJFQU1fSE9TVCA9ICJtaW5pbzE6OTAwMSIKcmFuZG9tLnNlZWQoIk5vIGxpc3Qgb2JqZWN0cy8xLjAiKQoKbGlzdF9yZXF1ZXN0X2NvdW50ZXIgPSAwCmxpc3RfcmVxdWVzdF9tYXhfbnVtYmVyID0gMTAKCgpkZWYgcmVxdWVzdChjb21tYW5kLCB1cmwsIGhlYWRlcnM9e30sIGRhdGE9Tm9uZSk6CiAgICAiIiJNaW5pLXJlcXVlc3RzLiIiIgoKICAgIGNsYXNzIER1bW15OgogICAgICAgIHBhc3MKCiAgICBwYXJ0cyA9IHVybGxpYi5wYXJzZS51cmxwYXJzZSh1cmwpCiAgICBjID0gaHR0cC5jbGllbnQuSFRUUENvbm5lY3Rpb24ocGFydHMuaG9zdG5hbWUsIHBhcnRzLnBvcnQpCiAgICBjLnJlcXVlc3QoCiAgICAgICAgY29tbWFuZCwKICAgICAgICB1cmxsaWIucGFyc2UudXJsdW5wYXJzZShwYXJ0cy5fcmVwbGFjZShzY2hlbWU9IiIsIG5ldGxvYz0iIikpLAogICAgICAgIGhlYWRlcnM9aGVhZGVycywKICAgICAgICBib2R5PWRhdGEsCiAgICApCiAgICByID0gYy5nZXRyZXNwb25zZSgpCiAgICByZXN1bHQgPSBEdW1teSgpCiAgICByZXN1bHQuc3RhdHVzX2NvZGUgPSByLnN0YXR1cwogICAgcmVzdWx0LmhlYWRlcnMgPSByLmhlYWRlcnMKICAgIHJlc3VsdC5jb250ZW50ID0gci5yZWFkKCkKICAgIHJldHVybiByZXN1bHQKCgpjbGFzcyBSZXF1ZXN0SGFuZGxlcihodHRwLnNlcnZlci5CYXNlSFRUUFJlcXVlc3RIYW5kbGVyKToKICAgIGRlZiBkb19HRVQoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoID09ICIvIjoKICAgICAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDIwMCkKICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQvcGxhaW4iKQogICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICAgICAgc2VsZi53ZmlsZS53cml0ZShiIk9LIikKICAgICAgICAgICAgcmV0dXJuCgogICAgICAgIHF1ZXJ5ID0gdXJsbGliLnBhcnNlLnVybHBhcnNlKHNlbGYucGF0aCkucXVlcnkKICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCgogICAgICAgIGdsb2JhbCBsaXN0X3JlcXVlc3RfY291bnRlcgogICAgICAgIGdsb2JhbCBsaXN0X3JlcXVlc3RfbWF4X251bWJlcgoKICAgICAgICBpZiAibGlzdC10eXBlIiBpbiBwYXJhbXM6CiAgICAgICAgICAgIGlmIGxpc3RfcmVxdWVzdF9jb3VudGVyID4gbGlzdF9yZXF1ZXN0X21heF9udW1iZXI6CiAgICAgICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNTAxKQogICAgICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgImFwcGxpY2F0aW9uL3htbCIpCiAgICAgICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoCiAgICAgICAgICAgICAgICAgICAgYiIiIjw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+CiAgICAgICAgICAgICAgICAgICAgICAgIDxFcnJvcj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxDb2RlPk5vdEltcGxlbWVudGVkPC9Db2RlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPE1lc3NhZ2U+SSBjYW4gbGlzdCBvYmplY3Qgb25seSBvbmNlPC9NZXNzYWdlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPFJlc291cmNlPlJFU09VUkNFPC9SZXNvdXJjZT4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxSZXF1ZXN0SWQ+UkVRVUVTVF9JRDwvUmVxdWVzdElkPgogICAgICAgICAgICAgICAgICAgICAgICA8L0Vycm9yPgogICAgICAgICAgICAgICAgICAgICIiIgogICAgICAgICAgICAgICAgKQogICAgICAgICAgICBlbHNlOgogICAgICAgICAgICAgICAgbGlzdF9yZXF1ZXN0X2NvdW50ZXIgKz0gMQogICAgICAgICAgICAgICAgc2VsZi5kb19IRUFEKCkKICAgICAgICBlbHNlOgogICAgICAgICAgICBzZWxmLmRvX0hFQUQoKQoKICAgIGRlZiBkb19QVVQoc2VsZik6CiAgICAgICAgc2VsZi5kb19IRUFEKCkKCiAgICBkZWYgZG9fREVMRVRFKHNlbGYpOgogICAgICAgIHNlbGYuZG9fSEVBRCgpCgogICAgZGVmIGRvX1BPU1Qoc2VsZik6CiAgICAgICAgZ2xvYmFsIGxpc3RfcmVxdWVzdF9jb3VudGVyCiAgICAgICAgZ2xvYmFsIGxpc3RfcmVxdWVzdF9tYXhfbnVtYmVyCgogICAgICAgIGlmIHNlbGYucGF0aC5zdGFydHN3aXRoKCIvcmVzZXRfY291bnRlcnMiKToKICAgICAgICAgICAgcXVlcnkgPSB1cmxsaWIucGFyc2UudXJscGFyc2Uoc2VsZi5wYXRoKS5xdWVyeQogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCgogICAgICAgICAgICBsaXN0X3JlcXVlc3RfY291bnRlciA9IDAKICAgICAgICAgICAgaWYgIm1heCIgaW4gcGFyYW1zOgogICAgICAgICAgICAgICAgbGlzdF9yZXF1ZXN0X21heF9udW1iZXIgPSBpbnQocGFyYW1zWyJtYXgiXVswXSkKCiAgICAgICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3BsYWluIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCiAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYiJPSyIpCiAgICAgICAgICAgIHJldHVybgoKICAgICAgICBzZWxmLmRvX0hFQUQoKQoKICAgIGRlZiBkb19IRUFEKHNlbGYpOgogICAgICAgIGNvbnRlbnRfbGVuZ3RoID0gc2VsZi5oZWFkZXJzLmdldCgiQ29udGVudC1MZW5ndGgiKQogICAgICAgIGRhdGEgPSBzZWxmLnJmaWxlLnJlYWQoaW50KGNvbnRlbnRfbGVuZ3RoKSkgaWYgY29udGVudF9sZW5ndGggZWxzZSBOb25lCiAgICAgICAgciA9IHJlcXVlc3QoCiAgICAgICAgICAgIHNlbGYuY29tbWFuZCwKICAgICAgICAgICAgZiJodHRwOi8ve1VQU1RSRUFNX0hPU1R9e3NlbGYucGF0aH0iLAogICAgICAgICAgICBoZWFkZXJzPXNlbGYuaGVhZGVycywKICAgICAgICAgICAgZGF0YT1kYXRhLAogICAgICAgICkKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2Uoci5zdGF0dXNfY29kZSkKICAgICAgICBmb3IgaywgdiBpbiByLmhlYWRlcnMuaXRlbXMoKToKICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcihrLCB2KQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoci5jb250ZW50KQogICAgICAgIHNlbGYud2ZpbGUuY2xvc2UoKQoKCmNsYXNzIFRocmVhZGVkSFRUUFNlcnZlcihzb2NrZXRzZXJ2ZXIuVGhyZWFkaW5nTWl4SW4sIGh0dHAuc2VydmVyLkhUVFBTZXJ2ZXIpOgogICAgIiIiSGFuZGxlIHJlcXVlc3RzIGluIGEgc2VwYXJhdGUgdGhyZWFkLiIiIgoKCmh0dHBkID0gVGhyZWFkZWRIVFRQU2VydmVyKCgiMC4wLjAuMCIsIGludChzeXMuYXJndlsxXSkpLCBSZXF1ZXN0SGFuZGxlcikKaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > no_list_objects.py'] (cluster.py:105, run_and_check)
2026-10-27 17:42:52 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:True nothrow:False cmd: ['python', 'no_list_objects.py', '8083'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:52 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:52 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8080/'] (cluster.py:105, run_and_check)
2026-10-27 17:42:53 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:42:53 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:53 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:42:54 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:42:54 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8082/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:54 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8082/'] (cluster.py:105, run_and_check)
2026-10-27 17:42:54 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:42:54 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:54 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8083/'] (cluster.py:105, run_and_check)
2026-10-27 17:42:55 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:42:56 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:56 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8080/'] (cluster.py:105, run_and_check)
2026-10-27 17:42:56 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:42:56 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:56 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:42:57 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:42:57 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8082/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:42:57 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8082/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:00 [ 373 ] DEBUG : Stdout:OK (cluster.py:113, run_and_check)
2026-10-27 17:43:00 [ 373 ] DEBUG : echo.py answered OK on attempt 2 (mock_servers.py:53, start_mock_servers)
2026-10-27 17:43:00 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:00 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8083/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:03 [ 373 ] DEBUG : Stdout:OK (cluster.py:113, run_and_check)
2026-10-27 17:43:03 [ 373 ] DEBUG : no_list_objects.py answered OK on attempt 2 (mock_servers.py:53, start_mock_servers)
2026-10-27 17:43:04 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:04 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8080/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:05 [ 373 ] DEBUG : Stdout:OK (cluster.py:113, run_and_check)
2026-10-27 17:43:05 [ 373 ] DEBUG : mock_s3.py answered OK on attempt 3 (mock_servers.py:53, start_mock_servers)
2026-10-27 17:43:05 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:05 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:05 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:06 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:06 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:06 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:07 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:07 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:08 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:09 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:09 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:09 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:10 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:10 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:11 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:12 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:12 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:12 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:13 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:13 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:13 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:14 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:14 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:15 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:16 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:16 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:16 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:17 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:17 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:17 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:18 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:18 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:19 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:20 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:20 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:20 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:21 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:21 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:21 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:22 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:22 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:23 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:24 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:24 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:24 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:25 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:25 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:25 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:26 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:26 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:27 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:28 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:28 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:28 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:29 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:29 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:29 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:30 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:30 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:31 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:32 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:32 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:32 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:33 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:33 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:33 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:34 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:34 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:35 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:36 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:36 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:36 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:37 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:37 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:37 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:38 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:38 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:39 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:40 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:40 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:40 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:41 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:41 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:41 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:42 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:42 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:43 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:44 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:44 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:44 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:45 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:45 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:45 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:46 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:46 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:46 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:47 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:47 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:48 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:49 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:49 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:49 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:50 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:50 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:50 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:51 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:51 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:52 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:53 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:53 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:53 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:54 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:54 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:54 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:55 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:55 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:55 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:56 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:56 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:57 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:58 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:58 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:58 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:43:59 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:43:59 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:43:59 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:00 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:00 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:01 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:02 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:02 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:02 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:03 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:03 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:03 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:04 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:04 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:05 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:06 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:06 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:06 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:07 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:07 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:07 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:08 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:08 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:08 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:09 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:09 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:10 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:11 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:11 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:11 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:12 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:12 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:12 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:13 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:13 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:13 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:14 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:14 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:15 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:16 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:16 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:16 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:17 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:17 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:17 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:18 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:18 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:19 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:20 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:20 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:20 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:21 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:21 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:21 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:22 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:22 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:23 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:24 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:24 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:24 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:25 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:25 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:25 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:26 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:26 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:26 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:27 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:27 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:28 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:29 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:29 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:29 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:30 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:30 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:30 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:31 [ 373 ] DEBUG : run container_id:rootteststorages3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:44:31 [ 373 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:44:32 [ 373 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:44:32 [ 373 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', '--project-name', 'rootteststorages3', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_non_default/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_with_environment_credentials/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:105, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_s3_max_redirects_1                ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_dummy_without_named_collections_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_dummy_1                           ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_s3_non_default_1                  ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_restricted_dummy_1                ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_s3_with_environment_credentials_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_minio1_1                          ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_resolver_1                        ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_proxy1_1                          ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_proxy2_1                          ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_dummy_1                           ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_s3_non_default_1                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_s3_with_environment_credentials_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_dummy_without_named_collections_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_restricted_dummy_1                ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_s3_max_redirects_1                ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_minio1_1                          ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_resolver_1                        ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_proxy1_1                          ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Stderr:Stopping rootteststorages3_proxy2_1                          ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/logs/stderr.log* || true'] (cluster.py:105, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/logs/stderr.log* || true'] (cluster.py:105, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/logs/stderr.log* || true'] (cluster.py:105, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/logs/stderr.log* || true'] (cluster.py:105, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_non_default/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_non_default/logs/stderr.log* || true'] (cluster.py:105, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_with_environment_credentials/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_with_environment_credentials/logs/stderr.log* || true'] (cluster.py:105, run_and_check)
2026-10-27 17:44:53 [ 373 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/.env', '--project-name', 'rootteststorages3', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/restricted_dummy/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/dummy_without_named_collections/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_max_redirects/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_non_default/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_0/s3_with_environment_credentials/docker-compose.yml', 'down', '--volumes'] (cluster.py:105, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_s3_max_redirects_1                ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_dummy_without_named_collections_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_dummy_1                           ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_s3_non_default_1                  ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_restricted_dummy_1                ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_s3_with_environment_credentials_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_minio1_1                          ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_resolver_1                        ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_proxy1_1                          ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_proxy2_1                          ...  (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_restricted_dummy_1                ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_resolver_1                        ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_s3_non_default_1                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_dummy_without_named_collections_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_proxy1_1                          ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_minio1_1                          ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_s3_max_redirects_1                ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_dummy_1                           ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_proxy2_1                          ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing rootteststorages3_s3_with_environment_credentials_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing network rootteststorages3_default (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stderr:Removing volume rootteststorages3_data1-1 (cluster.py:115, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Cleanup called (cluster.py:754, cleanup)
2026-10-27 17:44:54 [ 373 ] DEBUG : !!! Docker info: Client: Docker Engine - Community
 Version:    29.9.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 15
  Running: 12
  Paused: 1
  Stopped: 2
 Images: 38
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b7a25261e029d68978bbf290df0a62fc819b3c51
 runc version: v1.5.2-0-g29dd3dc2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-198-generic
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.33GiB
 Name: 02b61c2481ca
 ID: cbb4a713-cdad-4d08-9907-2668eda59ee0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  65.108.242.32:5000
  127.0.0.0/8
 Registry Mirrors:
  http://65.108.242.32:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 24

 (cluster.py:722, print_all_docker_pieces)
2026-10-27 17:44:54 [ 373 ] DEBUG : Docker networks for project rootteststorages3 are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:733, print_all_docker_pieces)
2026-10-27 17:44:54 [ 373 ] DEBUG : Docker containers for project rootteststorages3 are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:741, print_all_docker_pieces)
2026-10-27 17:44:54 [ 373 ] DEBUG : Docker volumes for project rootteststorages3 are DRIVER    VOLUME NAME
 (cluster.py:749, print_all_docker_pieces)
2026-10-27 17:44:54 [ 373 ] DEBUG : Command:docker container list --all --filter name='^/rootteststorages3_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup)
2026-10-27 17:44:54 [ 373 ] DEBUG : No running containers for project: rootteststorages3 (cluster.py:782, cleanup)
2026-10-27 17:44:54 [ 373 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup)
2026-10-27 17:44:54 [ 373 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup)
2026-10-27 17:44:54 [ 373 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Images pruned (cluster.py:807, cleanup)
2026-10-27 17:44:54 [ 373 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup)
2026-10-27 17:44:54 [ 373 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check)
2026-10-27 17:44:54 [ 373 ] DEBUG : Stdout:9 (cluster.py:113, run_and_check)
______________ ERROR at setup of test_create_new_files_on_insert _______________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
__________________ ERROR at setup of test_custom_auth_headers __________________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
_____________ ERROR at setup of test_custom_auth_headers_exclusion _____________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
______________________ ERROR at setup of test_empty_file _______________________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
___________________ ERROR at setup of test_empty_put[minio] ____________________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
________________ ERROR at setup of test_environment_credentials ________________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
_______________ ERROR at setup of test_filtering_by_file_or_path _______________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
___________________ ERROR at setup of test_format_detection ____________________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
______________ ERROR at setup of test_get_file_with_special[plus] ______________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
_____________ ERROR at setup of test_get_file_with_special[space] ______________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
_____________ ERROR at setup of test_get_path_with_special[plus2] ______________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
______________ ERROR at setup of test_get_path_with_special[plus] ______________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
_____________ ERROR at setup of test_get_path_with_special[space] ______________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
___________________ ERROR at setup of test_infinite_redirect ___________________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
____________ ERROR at setup of test_insert_select_schema_inference _____________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
______________ ERROR at setup of test_insert_with_path_with_globs ______________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
__________________ ERROR at setup of test_multipart[negative] __________________
[gw0] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "restricted_dummy",
                main_configs=["configs/config_for_test_remote_host_filter.xml"],
                with_minio=True,
            )
            cluster.add_instance(
                "dummy",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml", "configs/users.xml"],
            )
            cluster.add_instance(
                "dummy_without_named_collections",
                with_minio=True,
                main_configs=[
                    "configs/defaultS3.xml",
                    "configs/named_collections.xml",
                    "configs/schema_cache.xml",
                ],
                user_configs=["configs/access.xml"],
            )
            cluster.add_instance(
                "s3_max_redirects",
                with_minio=True,
                main_configs=["configs/defaultS3.xml"],
                user_configs=["configs/s3_max_redirects.xml"],
            )
            cluster.add_instance(
                "s3_non_default",
                with_minio=True,
            )
            cluster.add_instance(
                "s3_with_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "minio",
                    "AWS_SECRET_ACCESS_KEY": "minio123",
                },
                main_configs=["configs/use_environment_credentials.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test.py:661: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fae2db4cdc0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
=================================== FAILURES ===================================
______________________________ test_alias_columns ______________________________
[gw2] linux -- Python 3.10.12 /usr/bin/python3

    def test_alias_columns():
        instance.query(
            "CREATE TABLE table1(x Int32, y Int32, s Int32 ALIAS x + y) ENGINE = MergeTree ORDER BY tuple()"
        )
    
        select_query = "SELECT * FROM table1"
        assert (
            "it's necessary to have the grant SELECT(x, y) ON default.table1"
            in instance.query_and_get_error(select_query, user="A")
        )
    
        instance.query("GRANT SELECT(x,y) ON default.table1 TO A")
        assert instance.query(select_query, user="A") == ""
    
        select_query = "SELECT s FROM table1"
>       assert (
            "it's necessary to have the grant SELECT(s) ON default.table1"
            in instance.query_and_get_error(select_query, user="A")
        )
E       assert "it's necessary to have the grant SELECT(s) ON default.table1" in "Received exception from server (version 23.8.16):\nCode: 497. DB::Exception: Received from 172.16.7.2:9000. DB::Exception: A: Not enough privileges. To execute this query, it's necessary to have the grant SELECT(x, y, s) ON default.table1. Stack trace:\n\n0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e024ad3 in /usr/bin/clickhouse\n1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5b7590 in /usr/bin/clickhouse\n2. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000bd745d6 in /usr/bin/clickhouse\n3. ./build_docker/./src/Access/ContextAccess.cpp:520: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&, std::vector<String, std::allocator<String>> const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<String>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000002e0a9d71 in /usr/bin/clickhouse\n4. ./build_docker/./src/Access/ContextAccess.cpp:560: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&, std::vector<String, std::allocator<String>> const&) const @ 0x000000002e0a6a94 in /usr/bin/clickhouse\n5. ./build_docker/./src/Access/ContextAccess.cpp:631: DB::ContextAccess::checkAccess(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>> const&) const @ 0x000000002e01f25e in /usr/bin/clickhouse\n6. ./build_docker/./src/Interpreters/Context.cpp:1200: DB::Context::checkAccess(DB::AccessFlags const&, DB::StorageID const&, std::vector<String, std::allocator<String>> const&) const @ 0x000000002f924a0b in /usr/bin/clickhouse\n7. ./build_docker/./src/Planner/PlannerJoinTree.cpp:0: DB::buildJoinTreeQueryPlan(std::shared_ptr<DB::IQueryTreeNode> const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&, std::shared_ptr<DB::PlannerContext>&) @ 0x000000003163efc7 in /usr/bin/clickhouse\n8. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x00000000315f2736 in /usr/bin/clickhouse\n9. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000315ecaa7 in /usr/bin/clickhouse\n10. ./build_docker/./src/Planner/Planner.h:44: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x00000000315ea6f2 in /usr/bin/clickhouse\n11. ./build_docker/./src/Interpreters/executeQuery.cpp:1041: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320d82fd in /usr/bin/clickhouse\n12. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320cff81 in /usr/bin/clickhouse\n13. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ec12bd in /usr/bin/clickhouse\n14. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034efe0b0 in /usr/bin/clickhouse\n15. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dc58e2f in /usr/bin/clickhouse\n16. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dc59bf7 in /usr/bin/clickhouse\n17. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e13941c in /usr/bin/clickhouse\n18. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e1324c8 in /usr/bin/clickhouse\n19. ? @ 0x00007f42fdadfa83 in ?\n20. ? @ 0x00007f42fdb718e0 in ?\n. (ACCESS_DENIED)\n(query: SELECT s FROM table1)\n"
E        +  where "Received exception from server (version 23.8.16):\nCode: 497. DB::Exception: Received from 172.16.7.2:9000. DB::Exception: A: Not enough privileges. To execute this query, it's necessary to have the grant SELECT(x, y, s) ON default.table1. Stack trace:\n\n0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e024ad3 in /usr/bin/clickhouse\n1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5b7590 in /usr/bin/clickhouse\n2. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000bd745d6 in /usr/bin/clickhouse\n3. ./build_docker/./src/Access/ContextAccess.cpp:520: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&, std::vector<String, std::allocator<String>> const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<String>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000002e0a9d71 in /usr/bin/clickhouse\n4. ./build_docker/./src/Access/ContextAccess.cpp:560: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&, std::vector<String, std::allocator<String>> const&) const @ 0x000000002e0a6a94 in /usr/bin/clickhouse\n5. ./build_docker/./src/Access/ContextAccess.cpp:631: DB::ContextAccess::checkAccess(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>> const&) const @ 0x000000002e01f25e in /usr/bin/clickhouse\n6. ./build_docker/./src/Interpreters/Context.cpp:1200: DB::Context::checkAccess(DB::AccessFlags const&, DB::StorageID const&, std::vector<String, std::allocator<String>> const&) const @ 0x000000002f924a0b in /usr/bin/clickhouse\n7. ./build_docker/./src/Planner/PlannerJoinTree.cpp:0: DB::buildJoinTreeQueryPlan(std::shared_ptr<DB::IQueryTreeNode> const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&, std::shared_ptr<DB::PlannerContext>&) @ 0x000000003163efc7 in /usr/bin/clickhouse\n8. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x00000000315f2736 in /usr/bin/clickhouse\n9. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000315ecaa7 in /usr/bin/clickhouse\n10. ./build_docker/./src/Planner/Planner.h:44: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x00000000315ea6f2 in /usr/bin/clickhouse\n11. ./build_docker/./src/Interpreters/executeQuery.cpp:1041: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320d82fd in /usr/bin/clickhouse\n12. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320cff81 in /usr/bin/clickhouse\n13. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ec12bd in /usr/bin/clickhouse\n14. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034efe0b0 in /usr/bin/clickhouse\n15. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dc58e2f in /usr/bin/clickhouse\n16. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dc59bf7 in /usr/bin/clickhouse\n17. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e13941c in /usr/bin/clickhouse\n18. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e1324c8 in /usr/bin/clickhouse\n19. ? @ 0x00007f42fdadfa83 in ?\n20. ? @ 0x00007f42fdb718e0 in ?\n. (ACCESS_DENIED)\n(query: SELECT s FROM table1)\n" = <bound method ClickHouseInstance.query_and_get_error of <helpers.cluster.ClickHouseInstance object at 0x7f5deee958d0>>('SELECT s FROM table1', user='A')
E        +    where <bound method ClickHouseInstance.query_and_get_error of <helpers.cluster.ClickHouseInstance object at 0x7f5deee958d0>> = <helpers.cluster.ClickHouseInstance object at 0x7f5deee958d0>.query_and_get_error

test_select_access_rights/test_main.py:140: AssertionError
------------------------------ Captured log setup ------------------------------
2026-10-27 17:42:47 [ 379 ] DEBUG : Executing query CREATE USER OR REPLACE A on instance (cluster.py:3423, query)
------------------------------ Captured log call -------------------------------
2026-10-27 17:42:47 [ 379 ] DEBUG : Executing query CREATE TABLE table1(x Int32, y Int32, s Int32 ALIAS x + y) ENGINE = MergeTree ORDER BY tuple() on instance (cluster.py:3423, query)
2026-10-27 17:42:48 [ 379 ] DEBUG : Executing query SELECT * FROM table1 on instance (cluster.py:3495, query_and_get_error)
2026-10-27 17:42:48 [ 379 ] DEBUG : Executing query GRANT SELECT(x,y) ON default.table1 TO A on instance (cluster.py:3423, query)
2026-10-27 17:42:49 [ 379 ] DEBUG : Executing query SELECT * FROM table1 on instance (cluster.py:3423, query)
2026-10-27 17:42:49 [ 379 ] DEBUG : Executing query SELECT s FROM table1 on instance (cluster.py:3495, query_and_get_error)
---------------------------- Captured log teardown -----------------------------
2026-10-27 17:42:50 [ 379 ] DEBUG : Executing query DROP TABLE IF EXISTS table1 on instance (cluster.py:3423, query)
2026-10-27 17:42:50 [ 379 ] DEBUG : Executing query DROP TABLE IF EXISTS table2 on instance (cluster.py:3423, query)
______________________________ test_select_count _______________________________
[gw2] linux -- Python 3.10.12 /usr/bin/python3

    def test_select_count():
        instance.query(
            "CREATE TABLE table1(x String, y UInt8) ENGINE = MergeTree ORDER BY tuple()"
        )
    
        select_query = "SELECT count() FROM table1"
        assert (
            "it's necessary to have the grant SELECT for at least one column on default.table1"
            in instance.query_and_get_error(select_query, user="A")
        )
    
        instance.query("GRANT SELECT(x) ON default.table1 TO A")
>       assert instance.query(select_query, user="A") == "0\n"

test_select_access_rights/test_main.py:251: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
helpers/cluster.py:3424: in query
    return self.client.query(
helpers/client.py:36: in wrap
    return func(self, *args, **kwargs)
helpers/client.py:74: in query
    ).get_answer()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <helpers.client.CommandRequest object at 0x7f5dee7389a0>

    def get_answer(self):
        self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT)
        self.stdout_file.seek(0)
        self.stderr_file.seek(0)
    
        stdout = self.stdout_file.read().decode("utf-8", errors="replace")
        stderr = self.stderr_file.read().decode("utf-8", errors="replace")
    
        if (
            self.timer is not None
            and not self.process_finished_before_timeout
            and not self.ignore_error
        ):
            logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}")
            raise QueryTimeoutExceedException("Client timed out!")
    
        if (
            self.process.returncode != 0 or self.remove_trash_from_stderr(stderr)
        ) and not self.ignore_error:
>           raise QueryRuntimeException(
                "Client failed! Return code: {}, stderr: {}".format(
                    self.process.returncode, stderr
                ),
                self.process.returncode,
                stderr,
            )
E           helpers.client.QueryRuntimeException: Client failed! Return code: 241, stderr: Received exception from server (version 23.8.16):
E           Code: 497. DB::Exception: Received from 172.16.7.2:9000. DB::Exception: A: Not enough privileges. To execute this query, it's necessary to have the grant SELECT(y) ON default.table1. Stack trace:
E           
E           0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e024ad3 in /usr/bin/clickhouse
E           1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5b7590 in /usr/bin/clickhouse
E           2. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000bd745d6 in /usr/bin/clickhouse
E           3. ./build_docker/./src/Access/ContextAccess.cpp:520: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&, std::vector<String, std::allocator<String>> const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<String>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000002e0a9d71 in /usr/bin/clickhouse
E           4. ./build_docker/./src/Access/ContextAccess.cpp:560: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&, std::vector<String, std::allocator<String>> const&) const @ 0x000000002e0a6a94 in /usr/bin/clickhouse
E           5. ./build_docker/./src/Access/ContextAccess.cpp:631: DB::ContextAccess::checkAccess(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>> const&) const @ 0x000000002e01f25e in /usr/bin/clickhouse
E           6. ./build_docker/./src/Interpreters/Context.cpp:1200: DB::Context::checkAccess(DB::AccessFlags const&, DB::StorageID const&, std::vector<String, std::allocator<String>> const&) const @ 0x000000002f924a0b in /usr/bin/clickhouse
E           7. ./build_docker/./src/Planner/PlannerJoinTree.cpp:0: DB::buildJoinTreeQueryPlan(std::shared_ptr<DB::IQueryTreeNode> const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&, std::shared_ptr<DB::PlannerContext>&) @ 0x000000003163efc7 in /usr/bin/clickhouse
E           8. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x00000000315f2736 in /usr/bin/clickhouse
E           9. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000315ecaa7 in /usr/bin/clickhouse
E           10. ./build_docker/./src/Planner/PlannerJoinTree.cpp:0: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr<DB::IQueryTreeNode>, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::PlannerContext>&, bool, bool) @ 0x0000000031651e13 in /usr/bin/clickhouse
E           11. ./build_docker/./src/Planner/PlannerJoinTree.cpp:1450: DB::buildJoinTreeQueryPlan(std::shared_ptr<DB::IQueryTreeNode> const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&, std::shared_ptr<DB::PlannerContext>&) @ 0x000000003164160a in /usr/bin/clickhouse
E           12. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x00000000315f2736 in /usr/bin/clickhouse
E           13. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000315ecaa7 in /usr/bin/clickhouse
E           14. ./build_docker/./src/Planner/Planner.h:44: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x00000000315ea6f2 in /usr/bin/clickhouse
E           15. ./build_docker/./src/Interpreters/executeQuery.cpp:1041: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320d82fd in /usr/bin/clickhouse
E           16. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320cff81 in /usr/bin/clickhouse
E           17. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ec12bd in /usr/bin/clickhouse
E           18. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034efe0b0 in /usr/bin/clickhouse
E           19. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dc58e2f in /usr/bin/clickhouse
E           20. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dc59bf7 in /usr/bin/clickhouse
E           21. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e13941c in /usr/bin/clickhouse
E           22. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e1324c8 in /usr/bin/clickhouse
E           23. ? @ 0x00007f42fdadfa83 in ?
E           24. ? @ 0x00007f42fdb718e0 in ?
E           . (ACCESS_DENIED)
E           (query: SELECT count() FROM table1)

helpers/client.py:239: QueryRuntimeException
------------------------------ Captured log setup ------------------------------
2026-10-27 17:43:01 [ 379 ] DEBUG : Executing query CREATE USER OR REPLACE A on instance (cluster.py:3423, query)
------------------------------ Captured log call -------------------------------
2026-10-27 17:43:01 [ 379 ] DEBUG : Executing query CREATE TABLE table1(x String, y UInt8) ENGINE = MergeTree ORDER BY tuple() on instance (cluster.py:3423, query)
2026-10-27 17:43:01 [ 379 ] DEBUG : Executing query SELECT count() FROM table1 on instance (cluster.py:3495, query_and_get_error)
2026-10-27 17:43:02 [ 379 ] DEBUG : Executing query GRANT SELECT(x) ON default.table1 TO A on instance (cluster.py:3423, query)
2026-10-27 17:43:02 [ 379 ] DEBUG : Executing query SELECT count() FROM table1 on instance (cluster.py:3423, query)
---------------------------- Captured log teardown -----------------------------
2026-10-27 17:43:02 [ 379 ] DEBUG : Executing query DROP TABLE IF EXISTS table1 on instance (cluster.py:3423, query)
2026-10-27 17:43:03 [ 379 ] DEBUG : Executing query DROP TABLE IF EXISTS table2 on instance (cluster.py:3423, query)
_______________________________ test_select_join _______________________________
[gw2] linux -- Python 3.10.12 /usr/bin/python3

    def test_select_join():
        instance.query(
            "CREATE TABLE table1(d DATE, a String, b UInt8) ENGINE = MergeTree ORDER BY d"
        )
        instance.query(
            "CREATE TABLE table2(d DATE, x UInt32, y UInt8) ENGINE = MergeTree ORDER BY d"
        )
    
        select_query = "SELECT * FROM table1 JOIN table2 USING(d)"
>       assert (
            "it's necessary to have the grant SELECT(d, x, y) ON default.table2"
            in instance.query_and_get_error(select_query, user="A")
        )
E       assert "it's necessary to have the grant SELECT(d, x, y) ON default.table2" in "Received exception from server (version 23.8.16):\nCode: 497. DB::Exception: Received from 172.16.7.2:9000. DB::Exception: A: Not enough privileges. To execute this query, it's necessary to have the grant SELECT(d, a, b) ON default.table1. Stack trace:\n\n0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e024ad3 in /usr/bin/clickhouse\n1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5b7590 in /usr/bin/clickhouse\n2. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000bd745d6 in /usr/bin/clickhouse\n3. ./build_docker/./src/Access/ContextAccess.cpp:520: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&, std::vector<String, std::allocator<String>> const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<String>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000002e0a9d71 in /usr/bin/clickhouse\n4. ./build_docker/./src/Access/ContextAccess.cpp:560: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&, std::vector<String, std::allocator<String>> const&) const @ 0x000000002e0a6a94 in /usr/bin/clickhouse\n5. ./build_docker/./src/Access/ContextAccess.cpp:631: DB::ContextAccess::checkAccess(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>> const&) const @ 0x000000002e01f25e in /usr/bin/clickhouse\n6. ./build_docker/./src/Interpreters/Context.cpp:1200: DB::Context::checkAccess(DB::AccessFlags const&, DB::StorageID const&, std::vector<String, std::allocator<String>> const&) const @ 0x000000002f924a0b in /usr/bin/clickhouse\n7. ./build_docker/./src/Planner/PlannerJoinTree.cpp:0: DB::buildJoinTreeQueryPlan(std::shared_ptr<DB::IQueryTreeNode> const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&, std::shared_ptr<DB::PlannerContext>&) @ 0x000000003163efc7 in /usr/bin/clickhouse\n8. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x00000000315f2736 in /usr/bin/clickhouse\n9. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000315ecaa7 in /usr/bin/clickhouse\n10. ./build_docker/./src/Planner/Planner.h:44: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x00000000315ea6f2 in /usr/bin/clickhouse\n11. ./build_docker/./src/Interpreters/executeQuery.cpp:1041: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320d82fd in /usr/bin/clickhouse\n12. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320cff81 in /usr/bin/clickhouse\n13. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ec12bd in /usr/bin/clickhouse\n14. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034efe0b0 in /usr/bin/clickhouse\n15. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dc58e2f in /usr/bin/clickhouse\n16. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dc59bf7 in /usr/bin/clickhouse\n17. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e13941c in /usr/bin/clickhouse\n18. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e1324c8 in /usr/bin/clickhouse\n19. ? @ 0x00007f42fdadfa83 in ?\n20. ? @ 0x00007f42fdb718e0 in ?\n. (ACCESS_DENIED)\n(query: SELECT * FROM table1 JOIN table2 USING(d))\n"
E        +  where "Received exception from server (version 23.8.16):\nCode: 497. DB::Exception: Received from 172.16.7.2:9000. DB::Exception: A: Not enough privileges. To execute this query, it's necessary to have the grant SELECT(d, a, b) ON default.table1. Stack trace:\n\n0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e024ad3 in /usr/bin/clickhouse\n1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5b7590 in /usr/bin/clickhouse\n2. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000bd745d6 in /usr/bin/clickhouse\n3. ./build_docker/./src/Access/ContextAccess.cpp:520: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&, std::vector<String, std::allocator<String>> const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<String>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000002e0a9d71 in /usr/bin/clickhouse\n4. ./build_docker/./src/Access/ContextAccess.cpp:560: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&, std::vector<String, std::allocator<String>> const&) const @ 0x000000002e0a6a94 in /usr/bin/clickhouse\n5. ./build_docker/./src/Access/ContextAccess.cpp:631: DB::ContextAccess::checkAccess(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>> const&) const @ 0x000000002e01f25e in /usr/bin/clickhouse\n6. ./build_docker/./src/Interpreters/Context.cpp:1200: DB::Context::checkAccess(DB::AccessFlags const&, DB::StorageID const&, std::vector<String, std::allocator<String>> const&) const @ 0x000000002f924a0b in /usr/bin/clickhouse\n7. ./build_docker/./src/Planner/PlannerJoinTree.cpp:0: DB::buildJoinTreeQueryPlan(std::shared_ptr<DB::IQueryTreeNode> const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&, std::shared_ptr<DB::PlannerContext>&) @ 0x000000003163efc7 in /usr/bin/clickhouse\n8. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x00000000315f2736 in /usr/bin/clickhouse\n9. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000315ecaa7 in /usr/bin/clickhouse\n10. ./build_docker/./src/Planner/Planner.h:44: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x00000000315ea6f2 in /usr/bin/clickhouse\n11. ./build_docker/./src/Interpreters/executeQuery.cpp:1041: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320d82fd in /usr/bin/clickhouse\n12. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320cff81 in /usr/bin/clickhouse\n13. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ec12bd in /usr/bin/clickhouse\n14. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034efe0b0 in /usr/bin/clickhouse\n15. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dc58e2f in /usr/bin/clickhouse\n16. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dc59bf7 in /usr/bin/clickhouse\n17. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e13941c in /usr/bin/clickhouse\n18. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e1324c8 in /usr/bin/clickhouse\n19. ? @ 0x00007f42fdadfa83 in ?\n20. ? @ 0x00007f42fdb718e0 in ?\n. (ACCESS_DENIED)\n(query: SELECT * FROM table1 JOIN table2 USING(d))\n" = <bound method ClickHouseInstance.query_and_get_error of <helpers.cluster.ClickHouseInstance object at 0x7f5deee958d0>>('SELECT * FROM table1 JOIN table2 USING(d)', user='A')
E        +    where <bound method ClickHouseInstance.query_and_get_error of <helpers.cluster.ClickHouseInstance object at 0x7f5deee958d0>> = <helpers.cluster.ClickHouseInstance object at 0x7f5deee958d0>.query_and_get_error

test_select_access_rights/test_main.py:188: AssertionError
------------------------------ Captured log setup ------------------------------
2026-10-27 17:43:03 [ 379 ] DEBUG : Executing query CREATE USER OR REPLACE A on instance (cluster.py:3423, query)
------------------------------ Captured log call -------------------------------
2026-10-27 17:43:03 [ 379 ] DEBUG : Executing query CREATE TABLE table1(d DATE, a String, b UInt8) ENGINE = MergeTree ORDER BY d on instance (cluster.py:3423, query)
2026-10-27 17:43:03 [ 379 ] DEBUG : Executing query CREATE TABLE table2(d DATE, x UInt32, y UInt8) ENGINE = MergeTree ORDER BY d on instance (cluster.py:3423, query)
2026-10-27 17:43:04 [ 379 ] DEBUG : Executing query SELECT * FROM table1 JOIN table2 USING(d) on instance (cluster.py:3495, query_and_get_error)
---------------------------- Captured log teardown -----------------------------
2026-10-27 17:43:04 [ 379 ] DEBUG : Executing query DROP TABLE IF EXISTS table1 on instance (cluster.py:3423, query)
2026-10-27 17:43:04 [ 379 ] DEBUG : Executing query DROP TABLE IF EXISTS table2 on instance (cluster.py:3423, query)
______________________________ test_show_profiles ______________________________
[gw4] linux -- Python 3.10.12 /usr/bin/python3

    def test_show_profiles():
        instance.query("CREATE SETTINGS PROFILE xyz")
        assert instance.query("SHOW SETTINGS PROFILES") == "default\nreadonly\nxyz\n"
        assert instance.query("SHOW PROFILES") == "default\nreadonly\nxyz\n"
    
        assert instance.query("SHOW CREATE PROFILE xyz") == "CREATE SETTINGS PROFILE xyz\n"
>       assert (
            instance.query("SHOW CREATE SETTINGS PROFILE default")
            == "CREATE SETTINGS PROFILE default\n"
        )
E       AssertionError: assert 'CREATE SETTINGS PROFILE default SETTINGS allow_experimental_analyzer = true\n' == 'CREATE SETTINGS PROFILE default\n'
E         - CREATE SETTINGS PROFILE default
E         + CREATE SETTINGS PROFILE default SETTINGS allow_experimental_analyzer = true

test_settings_profile/test.py:457: AssertionError
------------------------------ Captured log call -------------------------------
2026-10-27 17:43:25 [ 385 ] DEBUG : Executing query CREATE SETTINGS PROFILE xyz on instance (cluster.py:3423, query)
2026-10-27 17:43:26 [ 385 ] DEBUG : Executing query SHOW SETTINGS PROFILES on instance (cluster.py:3423, query)
2026-10-27 17:43:26 [ 385 ] DEBUG : Executing query SHOW PROFILES on instance (cluster.py:3423, query)
2026-10-27 17:43:26 [ 385 ] DEBUG : Executing query SHOW CREATE PROFILE xyz on instance (cluster.py:3423, query)
2026-10-27 17:43:26 [ 385 ] DEBUG : Executing query SHOW CREATE SETTINGS PROFILE default on instance (cluster.py:3423, query)
---------------------------- Captured log teardown -----------------------------
2026-10-27 17:43:27 [ 385 ] DEBUG : Executing query CREATE USER OR REPLACE robin on instance (cluster.py:3423, query)
2026-10-27 17:43:27 [ 385 ] DEBUG : Executing query DROP ROLE IF EXISTS worker on instance (cluster.py:3423, query)
2026-10-27 17:43:27 [ 385 ] DEBUG : Executing query DROP SETTINGS PROFILE IF EXISTS xyz, alpha, P1, P2, P3, P4, P5, P6 on instance (cluster.py:3423, query)
_________________________ test_secure_connection[True] _________________________
[gw5] linux -- Python 3.10.12 /usr/bin/python3

started_cluster = <helpers.cluster.ClickHouseCluster object at 0x7fb9e0bf2ef0>

    @pytest.mark.parametrize("started_cluster", [True], indirect=["started_cluster"])
    def test_secure_connection(started_cluster):
        mongo_connection = get_mongo_connection(started_cluster, secure=True)
        db = mongo_connection["test"]
>       db.add_user("root", "clickhouse")

test_storage_mongodb/test.py:323: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.10/dist-packages/pymongo/database.py:1357: in add_user
    uinfo = self.command("usersInfo", name, session=session)
/usr/local/lib/python3.10/dist-packages/pymongo/database.py:736: in command
    with self.__client._socket_for_reads(
/usr/lib/python3.10/contextlib.py:135: in __enter__
    return next(self.gen)
/usr/local/lib/python3.10/dist-packages/pymongo/mongo_client.py:1325: in _socket_for_reads
    server = self._select_server(read_preference, session)
/usr/local/lib/python3.10/dist-packages/pymongo/mongo_client.py:1278: in _select_server
    server = topology.select_server(server_selector)
/usr/local/lib/python3.10/dist-packages/pymongo/topology.py:241: in select_server
    return random.choice(self.select_servers(selector,
/usr/local/lib/python3.10/dist-packages/pymongo/topology.py:199: in select_servers
    server_descriptions = self._select_servers_loop(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Topology <TopologyDescription id: 6ae0e2d4c85f7c034a986e59, topology_type: Single, servers: [<ServerDescription ('loc...st', 56531) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:56531: [Errno 111] Connection refused')>]>>
selector = Primary(), timeout = 30, address = None

    def _select_servers_loop(self, selector, timeout, address):
        """select_servers() guts. Hold the lock when calling this."""
        now = _time()
        end_time = now + timeout
        server_descriptions = self._description.apply_selector(
            selector, address, custom_selector=self._settings.server_selector)
    
        while not server_descriptions:
            # No suitable servers.
            if timeout == 0 or now > end_time:
>               raise ServerSelectionTimeoutError(
                    "%s, Timeout: %ss, Topology Description: %r" %
                    (self._error_message(selector), timeout, self.description))
E               pymongo.errors.ServerSelectionTimeoutError: localhost:56531: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 6ae0e2d4c85f7c034a986e59, topology_type: Single, servers: [<ServerDescription ('localhost', 56531) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:56531: [Errno 111] Connection refused')>]>

/usr/local/lib/python3.10/dist-packages/pymongo/topology.py:215: ServerSelectionTimeoutError
---------------------------- Captured stdout setup -----------------------------
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
---------------------------- Captured stderr setup -----------------------------
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
------------------------------ Captured log setup ------------------------------
2026-10-27 17:43:07 [ 388 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_mongodb/_instances_0/.env', '--project-name', 'rootteststoragemongodb', '--file', '/ClickHouse/tests/integration/test_storage_mongodb/_instances_0/node/docker-compose.yml', '--file', '/compose/docker_compose_mongo.yml', 'stop', '--timeout', '20'] (cluster.py:105, run_and_check)
2026-10-27 17:43:10 [ 388 ] DEBUG : Stderr:Stopping rootteststoragemongodb_node_1   ...  (cluster.py:115, run_and_check)
2026-10-27 17:43:10 [ 388 ] DEBUG : Stderr:Stopping rootteststoragemongodb_mongo2_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:43:10 [ 388 ] DEBUG : Stderr:Stopping rootteststoragemongodb_mongo1_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:43:10 [ 388 ] DEBUG : Stderr:Stopping rootteststoragemongodb_mongo1_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:43:10 [ 388 ] DEBUG : Stderr:Stopping rootteststoragemongodb_mongo2_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:43:10 [ 388 ] DEBUG : Stderr:Stopping rootteststoragemongodb_node_1   ... done (cluster.py:115, run_and_check)
2026-10-27 17:43:10 [ 388 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_storage_mongodb/_instances_0/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_mongodb/_instances_0/node/logs/stderr.log* || true'] (cluster.py:105, run_and_check)
2026-10-27 17:43:10 [ 388 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_mongodb/_instances_0/.env', '--project-name', 'rootteststoragemongodb', '--file', '/ClickHouse/tests/integration/test_storage_mongodb/_instances_0/node/docker-compose.yml', '--file', '/compose/docker_compose_mongo.yml', 'down', '--volumes'] (cluster.py:105, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : Stderr:Removing rootteststoragemongodb_node_1   ...  (cluster.py:115, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : Stderr:Removing rootteststoragemongodb_mongo2_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : Stderr:Removing rootteststoragemongodb_mongo1_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : Stderr:Removing rootteststoragemongodb_node_1   ... done (cluster.py:115, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : Stderr:Removing rootteststoragemongodb_mongo1_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : Stderr:Removing rootteststoragemongodb_mongo2_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : Stderr:Removing network rootteststoragemongodb_default (cluster.py:115, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : Cleanup called (cluster.py:754, cleanup)
2026-10-27 17:43:11 [ 388 ] DEBUG : !!! Docker info: Client: Docker Engine - Community
 Version:    29.9.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 36
  Running: 36
  Paused: 0
  Stopped: 0
 Images: 38
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b7a25261e029d68978bbf290df0a62fc819b3c51
 runc version: v1.5.2-0-g29dd3dc2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-198-generic
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.33GiB
 Name: 02b61c2481ca
 ID: cbb4a713-cdad-4d08-9907-2668eda59ee0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  65.108.242.32:5000
  127.0.0.0/8
 Registry Mirrors:
  http://65.108.242.32:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 24

 (cluster.py:722, print_all_docker_pieces)
2026-10-27 17:43:11 [ 388 ] DEBUG : Docker networks for project rootteststoragemongodb are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:733, print_all_docker_pieces)
2026-10-27 17:43:11 [ 388 ] DEBUG : Docker containers for project rootteststoragemongodb are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:741, print_all_docker_pieces)
2026-10-27 17:43:11 [ 388 ] DEBUG : Docker volumes for project rootteststoragemongodb are DRIVER    VOLUME NAME
 (cluster.py:749, print_all_docker_pieces)
2026-10-27 17:43:11 [ 388 ] DEBUG : Command:docker container list --all --filter name='^/rootteststoragemongodb_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup)
2026-10-27 17:43:11 [ 388 ] DEBUG : No running containers for project: rootteststoragemongodb (cluster.py:782, cleanup)
2026-10-27 17:43:11 [ 388 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup)
2026-10-27 17:43:11 [ 388 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup)
2026-10-27 17:43:11 [ 388 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : Images pruned (cluster.py:807, cleanup)
2026-10-27 17:43:11 [ 388 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup)
2026-10-27 17:43:11 [ 388 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : Stdout:11 (cluster.py:113, run_and_check)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                  DOCKER_KERBEROS_KDC_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV         CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                             MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                        JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                             TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                                 HOSTNAME 02b61c2481ca (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                                    SHLVL 0 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                                     HOME /root (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                                   OLDPWD / (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                        DOCKER_HELPER_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                         PYTHONUNBUFFERED 1 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                            UBSAN_OPTIONS print_stacktrace=1 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                           PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_role/test.py::test_admin_option test_role/test.py::test_changing_default_roles_affects_new_sessions_only test_role/test.py::test_combine_privileges test_role/test.py::test_create_role test_role/test.py::test_function_current_roles test_role/test.py::test_grant_role_to_role test_role/test.py::test_introspection test_role/test.py::test_revoke_requires_admin_option 'test_role/test.py::test_role_expiration[False]' 'test_role/test.py::test_role_expiration[True]' test_role/test.py::test_roles_cache test_role/test.py::test_set_role test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts test_select_access_rights/test_main.py::test_alias test_select_access_rights/test_main.py::test_alias_columns test_select_access_rights/test_main.py::test_materialized_columns test_select_access_rights/test_main.py::test_select_all_columns test_select_access_rights/test_main.py::test_select_all_columns_with_table_grant test_select_access_rights/test_main.py::test_select_count test_select_access_rights/test_main.py::test_select_join test_select_access_rights/test_main.py::test_select_prewhere test_select_access_rights/test_main.py::test_select_single_column test_select_access_rights/test_main.py::test_select_single_column_with_table_grant test_select_access_rights/test_main.py::test_select_union test_select_access_rights/test_main.py::test_select_where test_select_access_rights/test_main.py::test_select_with_row_policy 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6]' test_settings_profile/test.py::test_allow_ddl test_settings_profile/test.py::test_allow_introspection test_settings_profile/test.py::test_alter_and_drop test_settings_profile/test.py::test_changeable_in_readonly test_settings_profile/test.py::test_changing_default_profiles_affects_new_sessions_only test_settings_profile/test.py::test_function_current_profiles test_settings_profile/test.py::test_inheritance test_settings_profile/test.py::test_set_profile test_settings_profile/test.py::test_settings_aliases test_settings_profile/test.py::test_settings_from_granted_role test_settings_profile/test.py::test_show_profiles test_settings_profile/test.py::test_smoke test_sql_user_defined_functions_on_cluster/test.py::test_sql_user_defined_functions_on_cluster test_storage_delta/test.py::test_checkpoint test_storage_delta/test.py::test_metadata test_storage_delta/test.py::test_multiple_log_files test_storage_delta/test.py::test_partition_by test_storage_delta/test.py::test_single_log_file test_storage_delta/test.py::test_types test_storage_kerberized_kafka/test.py::test_kafka_json_as_string test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_no_kdc test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_request_new_ticket_after_expiration 'test_storage_meilisearch/test.py::test_incorrect_data_type[False]' 'test_storage_meilisearch/test.py::test_incorrect_data_type_secure[False]' 'test_storage_meilisearch/test.py::test_insert[False]' 'test_storage_meilisearch/test.py::test_insert_secure[False]' 'test_storage_meilisearch/test.py::test_meilimatch[False]' 'test_storage_meilisearch/test.py::test_meilimatch_secure[False]' 'test_storage_meilisearch/test.py::test_named_collection[False]' 'test_storage_meilisearch/test.py::test_named_collection_secure[False]' 'test_storage_meilisearch/test.py::test_security_levels[False]' 'test_storage_meilisearch/test.py::test_simple_select[False]' 'test_storage_meilisearch/test.py::test_simple_select_secure[False]' 'test_storage_meilisearch/test.py::test_table_function[False]' 'test_storage_meilisearch/test.py::test_table_function_secure[False]' 'test_storage_meilisearch/test.py::test_types[False]' 'test_storage_meilisearch/test.py::test_types_in_table_function[False]' 'test_storage_mongodb/test.py::test_arrays[False]' 'test_storage_mongodb/test.py::test_auth_source[False]' 'test_storage_mongodb/test.py::test_complex_data_type[False]' 'test_storage_mongodb/test.py::test_incorrect_data_type[False]' 'test_storage_mongodb/test.py::test_missing_columns[False]' 'test_storage_mongodb/test.py::test_no_credentials[False]' 'test_storage_mongodb/test.py::test_predefined_connection_configuration[False]' 'test_storage_mongodb/test.py::test_secure_connection[True]' 'test_storage_mongodb/test.py::test_simple_insert_select[False]' 'test_storage_mongodb/test.py::test_simple_select[False]' 'test_storage_mongodb/test.py::test_simple_select_from_view[False]' test_storage_s3/test.py::test_ast_auth_headers test_storage_s3/test.py::test_create_new_files_on_insert test_storage_s3/test.py::test_custom_auth_headers test_storage_s3/test.py::test_custom_auth_headers_exclusion test_storage_s3/test.py::test_empty_file 'test_storage_s3/test.py::test_empty_put[minio]' test_storage_s3/test.py::test_environment_credentials test_storage_s3/test.py::test_filtering_by_file_or_path test_storage_s3/test.py::test_format_detection 'test_storage_s3/test.py::test_get_file_with_special[plus]' 'test_storage_s3/test.py::test_get_file_with_special[space]' 'test_storage_s3/test.py::test_get_path_with_special[plus2]' 'test_storage_s3/test.py::test_get_path_with_special[plus]' 'test_storage_s3/test.py::test_get_path_with_special[space]' test_storage_s3/test.py::test_infinite_redirect test_storage_s3/test.py::test_insert_select_schema_inference test_storage_s3/test.py::test_insert_with_path_with_globs 'test_storage_s3/test.py::test_multipart[negative]'  -vvv (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV    CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                     COMPOSE_HTTP_TIMEOUT 600 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV              DOCKER_MYSQL_PHP_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                 DOCKER_DOTNET_CLIENT_TAG latest (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV         CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV               DOCKER_MYSQL_JS_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                                     PATH /spark-3.3.2-bin-hadoop3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV             DOCKER_KERBERIZED_HADOOP_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                           DOCKER_CHANNEL stable (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                    DOCKER_CLIENT_TIMEOUT 300 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV        DOCKER_POSTGRESQL_JAVA_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                     DOCKER_NGINX_DAV_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV           DOCKER_MYSQL_GOLANG_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV              CLICKHOUSE_USE_NEW_ANALYZER 1 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                                      PWD /ClickHouse/tests/integration (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV             DOCKER_MYSQL_JAVA_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV       CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV         CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                                       TZ Etc/UTC (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                                JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                          DOCKER_BASE_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                               SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                                 LC_CTYPE C.UTF-8 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                 INTEGRATION_TESTS_RUN_ID 0 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                  PYTEST_XDIST_TESTRUNUID 3f31a6e98a9b464fb8e4acb1a75d554d (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                      PYTEST_XDIST_WORKER gw5 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                PYTEST_XDIST_WORKER_COUNT 10 (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : ENV                      PYTEST_CURRENT_TEST test_storage_mongodb/test.py::test_secure_connection[True] (setup) (cluster.py:343, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:636, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : Removed :/ClickHouse/tests/integration/test_storage_mongodb/_instances_0 (cluster.py:639, __init__)
2026-10-27 17:43:11 [ 388 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1629, add_instance)
2026-10-27 17:43:11 [ 388 ] DEBUG : Cluster name: project_name:rootteststoragemongodb. Added instance name:node tag:1864 base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_mongodb/_instances_0/.env', '--project-name', 'rootteststoragemongodb', '--file', '/ClickHouse/tests/integration/test_storage_mongodb/_instances_0/node/docker-compose.yml', '--file', '/compose/docker_compose_mongo_secure.yml'] docker_compose_yml_dir:/compose (cluster.py:1915, add_instance)
2026-10-27 17:43:11 [ 388 ] INFO : Running tests in /ClickHouse/tests/integration/test_storage_mongodb/test.py (cluster.py:2637, start)
2026-10-27 17:43:11 [ 388 ] DEBUG : Cluster start called. is_up=False (cluster.py:2644, start)
2026-10-27 17:43:11 [ 388 ] DEBUG : !!! Docker info: Client: Docker Engine - Community
 Version:    29.9.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 36
  Running: 36
  Paused: 0
  Stopped: 0
 Images: 38
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b7a25261e029d68978bbf290df0a62fc819b3c51
 runc version: v1.5.2-0-g29dd3dc2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-198-generic
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.33GiB
 Name: 02b61c2481ca
 ID: cbb4a713-cdad-4d08-9907-2668eda59ee0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  65.108.242.32:5000
  127.0.0.0/8
 Registry Mirrors:
  http://65.108.242.32:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 24

 (cluster.py:722, print_all_docker_pieces)
2026-10-27 17:43:11 [ 388 ] DEBUG : Docker networks for project rootteststoragemongodb are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:733, print_all_docker_pieces)
2026-10-27 17:43:12 [ 388 ] DEBUG : Docker containers for project rootteststoragemongodb are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:741, print_all_docker_pieces)
2026-10-27 17:43:12 [ 388 ] DEBUG : Docker volumes for project rootteststoragemongodb are DRIVER    VOLUME NAME
 (cluster.py:749, print_all_docker_pieces)
2026-10-27 17:43:12 [ 388 ] DEBUG : Cleanup called (cluster.py:754, cleanup)
2026-10-27 17:43:12 [ 388 ] DEBUG : !!! Docker info: Client: Docker Engine - Community
 Version:    29.9.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 36
  Running: 36
  Paused: 0
  Stopped: 0
 Images: 38
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b7a25261e029d68978bbf290df0a62fc819b3c51
 runc version: v1.5.2-0-g29dd3dc2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-198-generic
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.33GiB
 Name: 02b61c2481ca
 ID: cbb4a713-cdad-4d08-9907-2668eda59ee0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  65.108.242.32:5000
  127.0.0.0/8
 Registry Mirrors:
  http://65.108.242.32:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 24

 (cluster.py:722, print_all_docker_pieces)
2026-10-27 17:43:12 [ 388 ] DEBUG : Docker networks for project rootteststoragemongodb are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:733, print_all_docker_pieces)
2026-10-27 17:43:12 [ 388 ] DEBUG : Docker containers for project rootteststoragemongodb are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:741, print_all_docker_pieces)
2026-10-27 17:43:12 [ 388 ] DEBUG : Docker volumes for project rootteststoragemongodb are DRIVER    VOLUME NAME
 (cluster.py:749, print_all_docker_pieces)
2026-10-27 17:43:12 [ 388 ] DEBUG : Command:docker container list --all --filter name='^/rootteststoragemongodb_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check)
2026-10-27 17:43:12 [ 388 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup)
2026-10-27 17:43:12 [ 388 ] DEBUG : No running containers for project: rootteststoragemongodb (cluster.py:782, cleanup)
2026-10-27 17:43:12 [ 388 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup)
2026-10-27 17:43:12 [ 388 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup)
2026-10-27 17:43:12 [ 388 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check)
2026-10-27 17:43:12 [ 388 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check)
2026-10-27 17:43:12 [ 388 ] DEBUG : Images pruned (cluster.py:807, cleanup)
2026-10-27 17:43:12 [ 388 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup)
2026-10-27 17:43:12 [ 388 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check)
2026-10-27 17:43:12 [ 388 ] DEBUG : Stdout:11 (cluster.py:113, run_and_check)
2026-10-27 17:43:12 [ 388 ] DEBUG : Setup directory for instance: node (cluster.py:2657, start)
2026-10-27 17:43:12 [ 388 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir)
2026-10-27 17:43:12 [ 388 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir)
2026-10-27 17:43:12 [ 388 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir)
2026-10-27 17:43:12 [ 388 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir)
2026-10-27 17:43:12 [ 388 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_mongodb/configs_secure/config.d/ssl_conf.xml', '/ClickHouse/tests/integration/test_storage_mongodb/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_storage_mongodb/_instances_0/node/configs/config.d (cluster.py:4383, create_dir)
2026-10-27 17:43:12 [ 388 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_mongodb/_instances_0/node/database (cluster.py:4400, create_dir)
2026-10-27 17:43:12 [ 388 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_mongodb/_instances_0/node/logs (cluster.py:4411, create_dir)
2026-10-27 17:43:12 [ 388 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir)
2026-10-27 17:43:12 [ 388 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'MONGO_HOST': 'mongo1', 'MONGO_EXTERNAL_PORT': '56531', 'MONGO_INTERNAL_PORT': '27017', 'MONGO_CONFIG_PATH': '/ClickHouse/tests/integration/helpers'} stored in /ClickHouse/tests/integration/test_storage_mongodb/_instances_0/.env (cluster.py:78, _create_env_file)
2026-10-27 17:43:12 [ 388 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file)
2026-10-27 17:43:12 [ 388 ] DEBUG : No config file found (config.py:28, find_config_file)
2026-10-27 17:43:12 [ 388 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file)
2026-10-27 17:43:12 [ 388 ] DEBUG : No config file found (config.py:28, find_config_file)
2026-10-27 17:43:12 [ 388 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 825 (connectionpool.py:546, _make_request)
2026-10-27 17:43:12 [ 388 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_mongodb/_instances_0/.env', '--project-name', 'rootteststoragemongodb', '--file', '/ClickHouse/tests/integration/test_storage_mongodb/_instances_0/node/docker-compose.yml', '--file', '/compose/docker_compose_mongo_secure.yml', 'pull'] (cluster.py:105, run_and_check)
2026-10-27 17:43:13 [ 388 ] DEBUG : Stderr:Pulling node   ...  (cluster.py:115, run_and_check)
2026-10-27 17:43:13 [ 388 ] DEBUG : Stderr:Pulling mongo1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:43:13 [ 388 ] DEBUG : Stderr:Pulling mongo1 ... pulling from library/mongo (cluster.py:115, run_and_check)
2026-10-27 17:43:13 [ 388 ] DEBUG : Stderr:Pulling node   ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:43:13 [ 388 ] DEBUG : Stderr:Pulling mongo1 ... digest: sha256:146c1fd999a660e697... (cluster.py:115, run_and_check)
2026-10-27 17:43:13 [ 388 ] DEBUG : Stderr:Pulling mongo1 ... status: image is up to date for m... (cluster.py:115, run_and_check)
2026-10-27 17:43:13 [ 388 ] DEBUG : Stderr:Pulling mongo1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:43:13 [ 388 ] DEBUG : Stderr:Pulling node   ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:43:13 [ 388 ] DEBUG : Stderr:Pulling node   ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:43:13 [ 388 ] DEBUG : Stderr:Pulling node   ... done (cluster.py:115, run_and_check)
2026-10-27 17:43:13 [ 388 ] DEBUG : Setup Mongo (cluster.py:2894, start)
2026-10-27 17:43:13 [ 388 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_mongodb/_instances_0/.env', '--project-name', 'rootteststoragemongodb', '--file', '/compose/docker_compose_mongo_secure.yml', '--verbose', 'up', '-d'] (cluster.py:105, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.config.config.find: Using configuration files: /compose/docker_compose_mongo_secure.yml (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.docker_client.get_client: docker-compose version 1.29.2, build unknown (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:docker-py version: <module 'docker.version' from '/usr/local/lib/python3.10/dist-packages/docker/version.py'> (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:CPython version: 3.10.12 (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:OpenSSL version: OpenSSL 3.0.2 15 Mar 2022 (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '23.0.6', 'Details': {'ApiVersion': '1.42', 'Arch': 'amd64', 'BuildTime': '2023-05-05T21:18:13.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9dbdbd4', 'GoVersion': 'go1.19.9', 'KernelVersion': '5.15.0-198-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': 'v2.4.2', 'Details': {'GitCommit': 'b7a25261e029d68978bbf290df0a62fc819b3c51'}}, {'Name': 'runc', 'Version': '1.5.2', 'Details': {'GitCommit': 'v1.5.2-0-g29dd3dc2'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=23.0.6, ApiVersion=1.42, MinAPIVersion=1.12, GitCommit=9dbdbd4, GoVersion=go1.19.9, Os=linux, Arch=amd64, KernelVersion=5.15.0-198-generic, BuildTime=2023-05-05T21:18:13.000000000+00:00 (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('rootteststoragemongodb_default') (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info <- () (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info -> {'Architecture': 'x86_64', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'BridgeNfIp6tables': True, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'BridgeNfIptables': True, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'CPUSet': True, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'CPUShares': True, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'CgroupDriver': 'cgroupfs', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'CgroupVersion': '2', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'ContainerdCommit': {'Expected': 'b7a25261e029d68978bbf290df0a62fc819b3c51', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:                      'ID': 'b7a25261e029d68978bbf290df0a62fc819b3c51'}, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Containers': 36, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('rootteststoragemongodb_default') (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.network.ensure: Creating network "rootteststoragemongodb_default" with the default driver (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network <- (name='rootteststoragemongodb_default', driver=None, options=None, ipam=None, internal=False, enable_ipv6=False, labels={'com.docker.compose.project': 'rootteststoragemongodb', 'com.docker.compose.network': 'default', 'com.docker.compose.version': '1.29.2'}, attachable=True, check_duplicate=True) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network -> {'Id': 'a9d4756ffabfc7875236051dd014352e4cec579b129f31f2f4a0b9509ac9c607', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Warning': ''} (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststoragemongodb', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststoragemongodb', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragemongodb', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragemongodb', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragemongodb', 'com.docker.compose.service=mongo1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragemongodb', 'com.docker.compose.service=mongo1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('mongo:3.6') (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Comment': '', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Cmd': ['mongod'], (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Entrypoint': ['docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragemongodb', 'com.docker.compose.service=mongo1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragemongodb', 'com.docker.compose.service=mongo1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: mongo1>} (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for <Service: mongo1> (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragemongodb', 'com.docker.compose.service=mongo1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragemongodb', 'com.docker.compose.service=mongo1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:Creating rootteststoragemongodb_mongo1_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststoragemongodb', service='mongo1', number=1)} (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststoragemongodb', service='mongo1', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('mongo:3.6') (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Comment': '', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Cmd': ['mongod'], (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Entrypoint': ['docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('mongo:3.6') (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Comment': '', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Cmd': ['mongod'], (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Entrypoint': ['docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: ac3ce2fa58d4e842f7014c90baab455e9b80b8764aa72f3ce73f09086c92e931 (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={'27017/tcp': ['56531']}, binds=['/ClickHouse/tests/integration/helpers:/mongo:rw'], volumes_from=[], privileged=False, network_mode='rootteststoragemongodb_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': ['/ClickHouse/tests/integration/helpers:/mongo:rw'], (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Links': [], (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'NetworkMode': 'rootteststoragemongodb_default', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'PortBindings': {'27017/tcp': [{'HostIp': '', 'HostPort': '56531'}]}, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'RestartPolicy': {'MaximumRetryCount': 0, 'Name': 'always'}, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (command='--config /mongo/mongo_secure.conf --profile=2 --verbose', environment=['MONGO_INITDB_ROOT_USERNAME=root', 'MONGO_INITDB_ROOT_PASSWORD=clickhouse'], image='mongo:3.6', ports=[('27017', 'tcp')], volumes={'/mongo': {}}, name='rootteststoragemongodb_mongo1_1', detach=True, labels={'com.docker.compose.project': 'rootteststoragemongodb', 'com.docker.compose.service': 'mongo1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/compose', 'com.docker.compose.project.config_files': '/compose/docker_compose_mongo_secure.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_mongodb/_instances_0/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': 'ac3ce2fa58d4e842f7014c90baab455e9b80b8764aa72f3ce73f09086c92e931'}, host_config={'NetworkMode': 'rootteststoragemongodb_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'VolumesFrom': [], 'Binds': ['/ClickHouse/tests/integration/helpers:/mongo:rw'], 'PortBindings': {'27017/tcp': [{'HostIp': '', 'HostPort': '56531'}]}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststoragemongodb_default': {'Aliases': ['mongo1'], 'IPAMConfig': {}}}}) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '56d420ee61c46cf9d3e423bda72870c6c066d954073b7ff87e2d220cd024f986', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('56d420ee61c46cf9d3e423bda72870c6c066d954073b7ff87e2d220cd024f986') (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Args': ['--config', '/mongo/mongo_secure.conf', '--profile=2', '--verbose'], (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Cmd': ['--config', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:                    '/mongo/mongo_secure.conf', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:                    '--profile=2', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:                    '--verbose'], (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('56d420ee61c46cf9d3e423bda72870c6c066d954073b7ff87e2d220cd024f986', 'rootteststoragemongodb_default') (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('56d420ee61c46cf9d3e423bda72870c6c066d954073b7ff87e2d220cd024f986', 'rootteststoragemongodb_default', aliases=['56d420ee61c4', 'mongo1'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('56d420ee61c46cf9d3e423bda72870c6c066d954073b7ff87e2d220cd024f986') (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststoragemongodb', service='mongo1', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:Creating rootteststoragemongodb_mongo1_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: <Service: mongo1> (cluster.py:115, run_and_check)
2026-10-27 17:43:15 [ 388 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:43:45 [ 388 ] DEBUG : Can't connect to Mongo localhost:56531: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 6ae0e2b3c85f7c034a986e58, topology_type: Single, servers: [<ServerDescription ('localhost', 56531) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:56531: [Errno 111] Connection refused')>]> (cluster.py:2455, wait_mongo_to_start)
2026-10-27 17:43:46 [ 388 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_storage_mongodb/_instances_0/.env --project-name rootteststoragemongodb --file /ClickHouse/tests/integration/test_storage_mongodb/_instances_0/node/docker-compose.yml --file /compose/docker_compose_mongo_secure.yml up -d --no-recreate') (cluster.py:2980, start)
2026-10-27 17:43:46 [ 388 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_mongodb/_instances_0/.env', '--project-name', 'rootteststoragemongodb', '--file', '/ClickHouse/tests/integration/test_storage_mongodb/_instances_0/node/docker-compose.yml', '--file', '/compose/docker_compose_mongo_secure.yml', 'up', '-d', '--no-recreate'] (cluster.py:105, run_and_check)
2026-10-27 17:43:47 [ 388 ] DEBUG : Stderr:Creating rootteststoragemongodb_node_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:43:47 [ 388 ] DEBUG : Stderr:Creating rootteststoragemongodb_node_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:43:47 [ 388 ] DEBUG : ClickHouse instance created (cluster.py:2988, start)
2026-10-27 17:43:47 [ 388 ] DEBUG : get_instance_ip instance_name=node (cluster.py:1971, get_instance_ip)
2026-10-27 17:43:47 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststoragemongodb_node_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:47 [ 388 ] DEBUG : Waiting for ClickHouse start in node, ip: 172.16.11.2... (cluster.py:2995, start)
2026-10-27 17:43:47 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststoragemongodb_node_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:47 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:47 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:47 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:47 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:47 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:47 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:47 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:47 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:48 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:48 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:48 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:48 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:48 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:48 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:48 [ 388 ] DEBUG : http://localhost:None "GET /v1.42/containers/7ef52807e94a9a17d40902db27256955d12f92b62fc145cf22323e80ff47b538/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:43:48 [ 388 ] DEBUG : ClickHouse node started (cluster.py:2999, start)
__________________ test_sql_user_defined_functions_on_cluster __________________
[gw9] linux -- Python 3.10.12 /usr/bin/python3

    def test_sql_user_defined_functions_on_cluster():
>       assert "Unknown function test_function" in ch1.query_and_get_error(
            "SELECT test_function(1);"
        )
E       assert 'Unknown function test_function' in "Received exception from server (version 23.8.16):\nCode: 46. DB::Exception: Received from 172.16.14.6:9000. DB::Exception: Function with name 'test_function' does not exists. In scope SELECT test_function(1). Stack trace:\n\n0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e024ad3 in /usr/bin/clickhouse\n1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5b7590 in /usr/bin/clickhouse\n2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<String&, String, String>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&, String&&, String&&) @ 0x000000002784a43b in /usr/bin/clickhouse\n3. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5064: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x00000000312670ca in /usr/bin/clickhouse\n4. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000003121ff0e in /usr/bin/clickhouse\n5. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5688: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000003121dc27 in /usr/bin/clickhouse\n6. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5877: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x0000000031237171 in /usr/bin/clickhouse\n7. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000003120aa94 in /usr/bin/clickhouse\n8. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>, std::shared_ptr<DB::Context const>) @ 0x0000000031204d56 in /usr/bin/clickhouse\n9. ./build_docker/./src/Analyzer/QueryTreePassManager.cpp:183: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x00000000311ffe68 in /usr/bin/clickhouse\n10. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:0: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x00000000315e7c52 in /usr/bin/clickhouse\n11. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:161: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&) @ 0x00000000315e3c69 in /usr/bin/clickhouse\n12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x0000000031435df5 in /usr/bin/clickhouse\n13. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320d7f1b in /usr/bin/clickhouse\n14. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320cff81 in /usr/bin/clickhouse\n15. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ec12bd in /usr/bin/clickhouse\n16. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034efe0b0 in /usr/bin/clickhouse\n17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dc58e2f in /usr/bin/clickhouse\n18. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dc59bf7 in /usr/bin/clickhouse\n19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e13941c in /usr/bin/clickhouse\n20. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e1324c8 in /usr/bin/clickhouse\n21. ? @ 0x00007ff41d9cda83 in ?\n22. ? @ 0x00007ff41da5f8e0 in ?\n. (UNKNOWN_FUNCTION)\n(query: SELECT test_function(1);)\n"
E        +  where "Received exception from server (version 23.8.16):\nCode: 46. DB::Exception: Received from 172.16.14.6:9000. DB::Exception: Function with name 'test_function' does not exists. In scope SELECT test_function(1). Stack trace:\n\n0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e024ad3 in /usr/bin/clickhouse\n1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5b7590 in /usr/bin/clickhouse\n2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<String&, String, String>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&, String&&, String&&) @ 0x000000002784a43b in /usr/bin/clickhouse\n3. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5064: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x00000000312670ca in /usr/bin/clickhouse\n4. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000003121ff0e in /usr/bin/clickhouse\n5. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5688: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000003121dc27 in /usr/bin/clickhouse\n6. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5877: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x0000000031237171 in /usr/bin/clickhouse\n7. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000003120aa94 in /usr/bin/clickhouse\n8. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>, std::shared_ptr<DB::Context const>) @ 0x0000000031204d56 in /usr/bin/clickhouse\n9. ./build_docker/./src/Analyzer/QueryTreePassManager.cpp:183: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x00000000311ffe68 in /usr/bin/clickhouse\n10. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:0: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x00000000315e7c52 in /usr/bin/clickhouse\n11. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:161: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&) @ 0x00000000315e3c69 in /usr/bin/clickhouse\n12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x0000000031435df5 in /usr/bin/clickhouse\n13. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320d7f1b in /usr/bin/clickhouse\n14. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320cff81 in /usr/bin/clickhouse\n15. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ec12bd in /usr/bin/clickhouse\n16. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034efe0b0 in /usr/bin/clickhouse\n17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dc58e2f in /usr/bin/clickhouse\n18. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dc59bf7 in /usr/bin/clickhouse\n19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e13941c in /usr/bin/clickhouse\n20. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e1324c8 in /usr/bin/clickhouse\n21. ? @ 0x00007ff41d9cda83 in ?\n22. ? @ 0x00007ff41da5f8e0 in ?\n. (UNKNOWN_FUNCTION)\n(query: SELECT test_function(1);)\n" = <bound method ClickHouseInstance.query_and_get_error of <helpers.cluster.ClickHouseInstance object at 0x7f6e3fa55f00>>('SELECT test_function(1);')
E        +    where <bound method ClickHouseInstance.query_and_get_error of <helpers.cluster.ClickHouseInstance object at 0x7f6e3fa55f00>> = <helpers.cluster.ClickHouseInstance object at 0x7f6e3fa55f00>.query_and_get_error

test_sql_user_defined_functions_on_cluster/test.py:27: AssertionError
---------------------------- Captured stdout setup -----------------------------
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
---------------------------- Captured stderr setup -----------------------------
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
------------------------------ Captured log setup ------------------------------
2026-10-27 17:46:29 [ 400 ] INFO : Running tests in /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/test.py (cluster.py:2637, start)
2026-10-27 17:46:29 [ 400 ] DEBUG : Cluster start called. is_up=False (cluster.py:2644, start)
2026-10-27 17:46:29 [ 400 ] DEBUG : !!! Docker info: Client: Docker Engine - Community
 Version:    29.9.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 38
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b7a25261e029d68978bbf290df0a62fc819b3c51
 runc version: v1.5.2-0-g29dd3dc2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-198-generic
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.33GiB
 Name: 02b61c2481ca
 ID: cbb4a713-cdad-4d08-9907-2668eda59ee0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  65.108.242.32:5000
  127.0.0.0/8
 Registry Mirrors:
  http://65.108.242.32:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 24

 (cluster.py:722, print_all_docker_pieces)
2026-10-27 17:46:29 [ 400 ] DEBUG : Docker networks for project roottestsqluserdefinedfunctionsoncluster are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:733, print_all_docker_pieces)
2026-10-27 17:46:29 [ 400 ] DEBUG : Docker containers for project roottestsqluserdefinedfunctionsoncluster are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:741, print_all_docker_pieces)
2026-10-27 17:46:29 [ 400 ] DEBUG : Docker volumes for project roottestsqluserdefinedfunctionsoncluster are DRIVER    VOLUME NAME
 (cluster.py:749, print_all_docker_pieces)
2026-10-27 17:46:29 [ 400 ] DEBUG : Cleanup called (cluster.py:754, cleanup)
2026-10-27 17:46:29 [ 400 ] DEBUG : !!! Docker info: Client: Docker Engine - Community
 Version:    29.9.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 38
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b7a25261e029d68978bbf290df0a62fc819b3c51
 runc version: v1.5.2-0-g29dd3dc2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-198-generic
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.33GiB
 Name: 02b61c2481ca
 ID: cbb4a713-cdad-4d08-9907-2668eda59ee0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  65.108.242.32:5000
  127.0.0.0/8
 Registry Mirrors:
  http://65.108.242.32:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 24

 (cluster.py:722, print_all_docker_pieces)
2026-10-27 17:46:29 [ 400 ] DEBUG : Docker networks for project roottestsqluserdefinedfunctionsoncluster are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:733, print_all_docker_pieces)
2026-10-27 17:46:29 [ 400 ] DEBUG : Docker containers for project roottestsqluserdefinedfunctionsoncluster are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:741, print_all_docker_pieces)
2026-10-27 17:46:29 [ 400 ] DEBUG : Docker volumes for project roottestsqluserdefinedfunctionsoncluster are DRIVER    VOLUME NAME
 (cluster.py:749, print_all_docker_pieces)
2026-10-27 17:46:29 [ 400 ] DEBUG : Command:docker container list --all --filter name='^/roottestsqluserdefinedfunctionsoncluster_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check)
2026-10-27 17:46:29 [ 400 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup)
2026-10-27 17:46:29 [ 400 ] DEBUG : No running containers for project: roottestsqluserdefinedfunctionsoncluster (cluster.py:782, cleanup)
2026-10-27 17:46:29 [ 400 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup)
2026-10-27 17:46:29 [ 400 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup)
2026-10-27 17:46:29 [ 400 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check)
2026-10-27 17:46:29 [ 400 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check)
2026-10-27 17:46:29 [ 400 ] DEBUG : Images pruned (cluster.py:807, cleanup)
2026-10-27 17:46:29 [ 400 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup)
2026-10-27 17:46:29 [ 400 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check)
2026-10-27 17:46:29 [ 400 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check)
2026-10-27 17:46:29 [ 400 ] DEBUG : Setup directory for instance: ch1 (cluster.py:2657, start)
2026-10-27 17:46:29 [ 400 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/configs/config.d/clusters.xml'] to /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch1/configs/config.d (cluster.py:4383, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch1/database (cluster.py:4400, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch1/logs (cluster.py:4411, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Setup directory for instance: ch2 (cluster.py:2657, start)
2026-10-27 17:46:29 [ 400 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/configs/config.d/clusters.xml'] to /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch2/configs/config.d (cluster.py:4383, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch2/database (cluster.py:4400, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch2/logs (cluster.py:4411, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Setup directory for instance: ch3 (cluster.py:2657, start)
2026-10-27 17:46:29 [ 400 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/configs/config.d/clusters.xml'] to /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch3/configs/config.d (cluster.py:4383, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch3/database (cluster.py:4400, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch3/logs (cluster.py:4411, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir)
2026-10-27 17:46:29 [ 400 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:1864', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/.env (cluster.py:78, _create_env_file)
2026-10-27 17:46:29 [ 400 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file)
2026-10-27 17:46:29 [ 400 ] DEBUG : No config file found (config.py:28, find_config_file)
2026-10-27 17:46:29 [ 400 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file)
2026-10-27 17:46:29 [ 400 ] DEBUG : No config file found (config.py:28, find_config_file)
2026-10-27 17:46:29 [ 400 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 825 (connectionpool.py:546, _make_request)
2026-10-27 17:46:29 [ 400 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/.env', '--project-name', 'roottestsqluserdefinedfunctionsoncluster', '--file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch1/docker-compose.yml', '--file', '/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch3/docker-compose.yml', 'pull'] (cluster.py:105, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo2 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo3 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch1  ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch3  ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch2  ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch1  ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo2 ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo3 ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch3  ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch2  ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch1  ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch1  ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch1  ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo3 ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo3 ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo2 ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo2 ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo3 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo2 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch3  ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch3  ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch2  ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch2  ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch3  ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling ch2  ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo1 ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo1 ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo1 ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Stderr:Pulling zoo1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:30 [ 400 ] DEBUG : Setup ZooKeeper (cluster.py:2699, start)
2026-10-27 17:46:30 [ 400 ] DEBUG : Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/log', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/config', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/coordination', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper2/log', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper2/config', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper2/coordination', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/log', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/config', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/coordination'] (cluster.py:2700, start)
2026-10-27 17:46:30 [ 400 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/.env', '--project-name', 'roottestsqluserdefinedfunctionsoncluster', '--file', '/compose/docker_compose_keeper.yml', '--verbose', 'up', '-d'] (cluster.py:105, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.config.config.find: Using configuration files: /compose/docker_compose_keeper.yml (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.docker_client.get_client: docker-compose version 1.29.2, build unknown (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:docker-py version: <module 'docker.version' from '/usr/local/lib/python3.10/dist-packages/docker/version.py'> (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:CPython version: 3.10.12 (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:OpenSSL version: OpenSSL 3.0.2 15 Mar 2022 (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '23.0.6', 'Details': {'ApiVersion': '1.42', 'Arch': 'amd64', 'BuildTime': '2023-05-05T21:18:13.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9dbdbd4', 'GoVersion': 'go1.19.9', 'KernelVersion': '5.15.0-198-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': 'v2.4.2', 'Details': {'GitCommit': 'b7a25261e029d68978bbf290df0a62fc819b3c51'}}, {'Name': 'runc', 'Version': '1.5.2', 'Details': {'GitCommit': 'v1.5.2-0-g29dd3dc2'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=23.0.6, ApiVersion=1.42, MinAPIVersion=1.12, GitCommit=9dbdbd4, GoVersion=go1.19.9, Os=linux, Arch=amd64, KernelVersion=5.15.0-198-generic, BuildTime=2023-05-05T21:18:13.000000000+00:00 (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('roottestsqluserdefinedfunctionsoncluster_default') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info <- () (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info -> {'Architecture': 'x86_64', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'BridgeNfIp6tables': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'BridgeNfIptables': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'CPUSet': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'CPUShares': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'CgroupDriver': 'cgroupfs', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'CgroupVersion': '2', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'ContainerdCommit': {'Expected': 'b7a25261e029d68978bbf290df0a62fc819b3c51', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:                      'ID': 'b7a25261e029d68978bbf290df0a62fc819b3c51'}, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Containers': 1, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('roottestsqluserdefinedfunctionsoncluster_default') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.network.ensure: Creating network "roottestsqluserdefinedfunctionsoncluster_default" with the default driver (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network <- (name='roottestsqluserdefinedfunctionsoncluster_default', driver=None, options=None, ipam=None, internal=False, enable_ipv6=False, labels={'com.docker.compose.project': 'roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.network': 'default', 'com.docker.compose.version': '1.29.2'}, attachable=True, check_duplicate=True) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network -> {'Id': '3f6c783a171e52e4a860e6254c34a6eb608f602730c040c94201ff4cf570b403', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Warning': ''} (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1864') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1864') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1864') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: zoo2>, <Service: zoo1>, <Service: zoo3>} (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for <Service: zoo2> (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for <Service: zoo1> (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for <Service: zoo3> (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:Creating roottestsqluserdefinedfunctionsoncluster_zoo2_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestsqluserdefinedfunctionsoncluster', service='zoo2', number=1)} (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestsqluserdefinedfunctionsoncluster', service='zoo2', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1864') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1864') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:Creating roottestsqluserdefinedfunctionsoncluster_zoo3_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestsqluserdefinedfunctionsoncluster', service='zoo3', number=1)} (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestsqluserdefinedfunctionsoncluster', service='zoo3', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1864') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 66e42819bef4281857578abcc85437a3db9b1e5b3c51e5f1e1fa82de1f200a90 (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestsqluserdefinedfunctionsoncluster_default', devices=None, device_requests=None, dns=None, dns_opt=['attempts:2', 'timeout:1', 'inet6', 'rotate'], dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=['label:disable'], ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=[{'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper2/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper2/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper2/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}], device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Links': [], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Mounts': [{'ReadOnly': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:             'Source': '/clickhouse', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:             'Target': '/usr/bin/clickhouse-keeper', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:             'Type': 'bind'}, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            {'ReadOnly': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (entrypoint='clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config2.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log', image='altinityinfra/integration-test:1864', user='0', volumes={}, name='roottestsqluserdefinedfunctionsoncluster_zoo2_1', detach=True, environment=[], labels={'com.docker.compose.project': 'roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service': 'zoo2', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/compose', 'com.docker.compose.project.config_files': '/compose/docker_compose_keeper.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '66e42819bef4281857578abcc85437a3db9b1e5b3c51e5f1e1fa82de1f200a90'}, host_config={'NetworkMode': 'roottestsqluserdefinedfunctionsoncluster_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], 'SecurityOpt': ['label:disable'], 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}, 'Mounts': [{'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper2/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper2/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper2/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}]}, networking_config={'EndpointsConfig': {'roottestsqluserdefinedfunctionsoncluster_default': {'Aliases': ['zoo2'], 'IPAMConfig': {}}}}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:Creating roottestsqluserdefinedfunctionsoncluster_zoo1_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestsqluserdefinedfunctionsoncluster', service='zoo1', number=1)} (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestsqluserdefinedfunctionsoncluster', service='zoo1', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1864') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1864') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1864') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 09f2bb981e68d9137686a5b93ebe95d5e072c94bb07cb2221261e8cb3632ecb9 (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestsqluserdefinedfunctionsoncluster_default', devices=None, device_requests=None, dns=None, dns_opt=['attempts:2', 'timeout:1', 'inet6', 'rotate'], dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=['label:disable'], ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=[{'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/config', 'Type': 'bind', 'ReadOnly': None}], device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Links': [], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Mounts': [{'ReadOnly': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:             'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/coordination', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:             'Target': '/var/lib/clickhouse-keeper', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:             'Type': 'bind'}, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            {'ReadOnly': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (entrypoint='clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config3.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log', image='altinityinfra/integration-test:1864', user='0', volumes={}, name='roottestsqluserdefinedfunctionsoncluster_zoo3_1', detach=True, environment=[], labels={'com.docker.compose.project': 'roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service': 'zoo3', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/compose', 'com.docker.compose.project.config_files': '/compose/docker_compose_keeper.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '09f2bb981e68d9137686a5b93ebe95d5e072c94bb07cb2221261e8cb3632ecb9'}, host_config={'NetworkMode': 'roottestsqluserdefinedfunctionsoncluster_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], 'SecurityOpt': ['label:disable'], 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}, 'Mounts': [{'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper3/config', 'Type': 'bind', 'ReadOnly': None}]}, networking_config={'EndpointsConfig': {'roottestsqluserdefinedfunctionsoncluster_default': {'Aliases': ['zoo3'], 'IPAMConfig': {}}}}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 692d29facce477a5d5b4ef8d04209e94bac3022b0de79fcb936ed16b40559f94 (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestsqluserdefinedfunctionsoncluster_default', devices=None, device_requests=None, dns=None, dns_opt=['attempts:2', 'timeout:1', 'inet6', 'rotate'], dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=['label:disable'], ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=[{'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/config', 'Type': 'bind', 'ReadOnly': None}], device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Links': [], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Mounts': [{'ReadOnly': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:             'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/log', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:             'Target': '/var/log/clickhouse-keeper', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:             'Type': 'bind'}, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            {'ReadOnly': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (entrypoint='clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config1.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log', image='altinityinfra/integration-test:1864', user='0', volumes={}, name='roottestsqluserdefinedfunctionsoncluster_zoo1_1', detach=True, environment=[], labels={'com.docker.compose.project': 'roottestsqluserdefinedfunctionsoncluster', 'com.docker.compose.service': 'zoo1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/compose', 'com.docker.compose.project.config_files': '/compose/docker_compose_keeper.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '692d29facce477a5d5b4ef8d04209e94bac3022b0de79fcb936ed16b40559f94'}, host_config={'NetworkMode': 'roottestsqluserdefinedfunctionsoncluster_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], 'SecurityOpt': ['label:disable'], 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}, 'Mounts': [{'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/keeper1/config', 'Type': 'bind', 'ReadOnly': None}]}, networking_config={'EndpointsConfig': {'roottestsqluserdefinedfunctionsoncluster_default': {'Aliases': ['zoo1'], 'IPAMConfig': {}}}}) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '4fa5084131cfdc07ba185a656b3293a808cd4ba172e79ad7e5505208b3cf17dd', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('4fa5084131cfdc07ba185a656b3293a808cd4ba172e79ad7e5505208b3cf17dd') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:          '--config=/etc/clickhouse-keeper/keeper_config2.xml', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:          '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:          '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Cmd': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('4fa5084131cfdc07ba185a656b3293a808cd4ba172e79ad7e5505208b3cf17dd', 'roottestsqluserdefinedfunctionsoncluster_default') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '54b5f72e9d99ab39d449a3b7d6bb3f42b1c30e8920839163e918ddea80d222a9', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('54b5f72e9d99ab39d449a3b7d6bb3f42b1c30e8920839163e918ddea80d222a9') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': 'be40580e9f3acfd72199c33182594b6c6c8bf10a13accdc45fe69e55b29536ef', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('be40580e9f3acfd72199c33182594b6c6c8bf10a13accdc45fe69e55b29536ef') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:          '--config=/etc/clickhouse-keeper/keeper_config1.xml', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:          '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:          '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Cmd': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('54b5f72e9d99ab39d449a3b7d6bb3f42b1c30e8920839163e918ddea80d222a9', 'roottestsqluserdefinedfunctionsoncluster_default') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:          '--config=/etc/clickhouse-keeper/keeper_config3.xml', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:          '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:          '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Cmd': None, (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('be40580e9f3acfd72199c33182594b6c6c8bf10a13accdc45fe69e55b29536ef', 'roottestsqluserdefinedfunctionsoncluster_default') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('4fa5084131cfdc07ba185a656b3293a808cd4ba172e79ad7e5505208b3cf17dd', 'roottestsqluserdefinedfunctionsoncluster_default', aliases=['zoo2', '4fa5084131cf'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('54b5f72e9d99ab39d449a3b7d6bb3f42b1c30e8920839163e918ddea80d222a9', 'roottestsqluserdefinedfunctionsoncluster_default', aliases=['54b5f72e9d99', 'zoo1'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('be40580e9f3acfd72199c33182594b6c6c8bf10a13accdc45fe69e55b29536ef', 'roottestsqluserdefinedfunctionsoncluster_default', aliases=['zoo3', 'be40580e9f3a'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('4fa5084131cfdc07ba185a656b3293a808cd4ba172e79ad7e5505208b3cf17dd') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('54b5f72e9d99ab39d449a3b7d6bb3f42b1c30e8920839163e918ddea80d222a9') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('be40580e9f3acfd72199c33182594b6c6c8bf10a13accdc45fe69e55b29536ef') (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestsqluserdefinedfunctionsoncluster', service='zoo2', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:Creating roottestsqluserdefinedfunctionsoncluster_zoo2_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: <Service: zoo2> (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestsqluserdefinedfunctionsoncluster', service='zoo1', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:Creating roottestsqluserdefinedfunctionsoncluster_zoo1_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: <Service: zoo1> (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestsqluserdefinedfunctionsoncluster', service='zoo3', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:Creating roottestsqluserdefinedfunctionsoncluster_zoo3_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: <Service: zoo3> (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:46:32 [ 400 ] DEBUG : Wait ZooKeeper to start (cluster.py:2337, wait_zookeeper_to_start)
2026-10-27 17:46:32 [ 400 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:1971, get_instance_ip)
2026-10-27 17:46:32 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestsqluserdefinedfunctionsoncluster_zoo1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:32 [ 400 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.14.4, port:2181, use_ssl:False (cluster.py:3120, get_kazoo_client)
2026-10-27 17:46:32 [ 400 ] INFO : Connecting to 172.16.14.4(172.16.14.4):2181, use_ssl: False (connection.py:650, _connect)
2026-10-27 17:46:32 [ 400 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt)
2026-10-27 17:46:32 [ 400 ] INFO : Connecting to 172.16.14.4(172.16.14.4):2181, use_ssl: False (connection.py:650, _connect)
2026-10-27 17:46:32 [ 400 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt)
2026-10-27 17:46:32 [ 400 ] INFO : Connecting to 172.16.14.4(172.16.14.4):2181, use_ssl: False (connection.py:650, _connect)
2026-10-27 17:46:32 [ 400 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt)
2026-10-27 17:46:33 [ 400 ] INFO : Connecting to 172.16.14.4(172.16.14.4):2181, use_ssl: False (connection.py:650, _connect)
2026-10-27 17:46:33 [ 400 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt)
2026-10-27 17:46:34 [ 400 ] INFO : Connecting to 172.16.14.4(172.16.14.4):2181, use_ssl: False (connection.py:650, _connect)
2026-10-27 17:46:34 [ 400 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt)
2026-10-27 17:46:35 [ 400 ] INFO : Connecting to 172.16.14.4(172.16.14.4):2181, use_ssl: False (connection.py:650, _connect)
2026-10-27 17:46:35 [ 400 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit)
2026-10-27 17:46:35 [ 400 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback)
2026-10-27 17:46:35 [ 400 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit)
2026-10-27 17:46:35 [ 400 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response)
2026-10-27 17:46:35 [ 400 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit)
2026-10-27 17:46:35 [ 400 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt)
2026-10-27 17:46:35 [ 400 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt)
2026-10-27 17:46:35 [ 400 ] INFO : Zookeeper connection lost (client.py:543, _session_callback)
2026-10-27 17:46:35 [ 400 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop)
2026-10-27 17:46:35 [ 400 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback)
2026-10-27 17:46:35 [ 400 ] DEBUG : get_instance_ip instance_name=zoo2 (cluster.py:1971, get_instance_ip)
2026-10-27 17:46:35 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestsqluserdefinedfunctionsoncluster_zoo2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:35 [ 400 ] DEBUG : get_kazoo_client: zoo2, ip:172.16.14.2, port:2181, use_ssl:False (cluster.py:3120, get_kazoo_client)
2026-10-27 17:46:35 [ 400 ] INFO : Connecting to 172.16.14.2(172.16.14.2):2181, use_ssl: False (connection.py:650, _connect)
2026-10-27 17:46:35 [ 400 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit)
2026-10-27 17:46:35 [ 400 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback)
2026-10-27 17:46:35 [ 400 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit)
2026-10-27 17:46:35 [ 400 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response)
2026-10-27 17:46:35 [ 400 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit)
2026-10-27 17:46:35 [ 400 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt)
2026-10-27 17:46:35 [ 400 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt)
2026-10-27 17:46:35 [ 400 ] INFO : Zookeeper connection lost (client.py:543, _session_callback)
2026-10-27 17:46:35 [ 400 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop)
2026-10-27 17:46:35 [ 400 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback)
2026-10-27 17:46:35 [ 400 ] DEBUG : get_instance_ip instance_name=zoo3 (cluster.py:1971, get_instance_ip)
2026-10-27 17:46:35 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestsqluserdefinedfunctionsoncluster_zoo3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:35 [ 400 ] DEBUG : get_kazoo_client: zoo3, ip:172.16.14.3, port:2181, use_ssl:False (cluster.py:3120, get_kazoo_client)
2026-10-27 17:46:35 [ 400 ] INFO : Connecting to 172.16.14.3(172.16.14.3):2181, use_ssl: False (connection.py:650, _connect)
2026-10-27 17:46:35 [ 400 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit)
2026-10-27 17:46:35 [ 400 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback)
2026-10-27 17:46:35 [ 400 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit)
2026-10-27 17:46:35 [ 400 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response)
2026-10-27 17:46:35 [ 400 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit)
2026-10-27 17:46:35 [ 400 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt)
2026-10-27 17:46:35 [ 400 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt)
2026-10-27 17:46:35 [ 400 ] INFO : Zookeeper connection lost (client.py:543, _session_callback)
2026-10-27 17:46:35 [ 400 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop)
2026-10-27 17:46:35 [ 400 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback)
2026-10-27 17:46:35 [ 400 ] DEBUG : All instances of ZooKeeper started (cluster.py:2345, wait_zookeeper_to_start)
2026-10-27 17:46:35 [ 400 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/.env --project-name roottestsqluserdefinedfunctionsoncluster --file /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch1/docker-compose.yml --file /compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch2/docker-compose.yml --file /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch3/docker-compose.yml up -d --no-recreate') (cluster.py:2980, start)
2026-10-27 17:46:35 [ 400 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/.env', '--project-name', 'roottestsqluserdefinedfunctionsoncluster', '--file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch1/docker-compose.yml', '--file', '/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch3/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:105, run_and_check)
2026-10-27 17:46:37 [ 400 ] DEBUG : Stderr:Creating roottestsqluserdefinedfunctionsoncluster_ch1_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:37 [ 400 ] DEBUG : Stderr:Creating roottestsqluserdefinedfunctionsoncluster_ch3_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:37 [ 400 ] DEBUG : Stderr:Creating roottestsqluserdefinedfunctionsoncluster_ch2_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:37 [ 400 ] DEBUG : Stderr:Creating roottestsqluserdefinedfunctionsoncluster_ch2_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:37 [ 400 ] DEBUG : Stderr:Creating roottestsqluserdefinedfunctionsoncluster_ch1_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:37 [ 400 ] DEBUG : Stderr:Creating roottestsqluserdefinedfunctionsoncluster_ch3_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:37 [ 400 ] DEBUG : ClickHouse instance created (cluster.py:2988, start)
2026-10-27 17:46:37 [ 400 ] DEBUG : get_instance_ip instance_name=ch1 (cluster.py:1971, get_instance_ip)
2026-10-27 17:46:37 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestsqluserdefinedfunctionsoncluster_ch1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:37 [ 400 ] DEBUG : Waiting for ClickHouse start in ch1, ip: 172.16.14.6... (cluster.py:2995, start)
2026-10-27 17:46:37 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestsqluserdefinedfunctionsoncluster_ch1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:37 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:37 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:37 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:37 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:37 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:37 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:37 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:37 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:37 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:38 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:38 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:38 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:38 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/b399303053230263a10ff9a9cf696219453c7b67879b56dc0adbcea3bd623ac4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:38 [ 400 ] DEBUG : ClickHouse ch1 started (cluster.py:2999, start)
2026-10-27 17:46:38 [ 400 ] DEBUG : get_instance_ip instance_name=ch2 (cluster.py:1971, get_instance_ip)
2026-10-27 17:46:38 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestsqluserdefinedfunctionsoncluster_ch2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:38 [ 400 ] DEBUG : Waiting for ClickHouse start in ch2, ip: 172.16.14.5... (cluster.py:2995, start)
2026-10-27 17:46:38 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestsqluserdefinedfunctionsoncluster_ch2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:38 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/ee13e539e230f6bfe0e5aa15a73c6162aa940d2070e3898eba35603bdff890f8/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:38 [ 400 ] DEBUG : ClickHouse ch2 started (cluster.py:2999, start)
2026-10-27 17:46:38 [ 400 ] DEBUG : get_instance_ip instance_name=ch3 (cluster.py:1971, get_instance_ip)
2026-10-27 17:46:38 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestsqluserdefinedfunctionsoncluster_ch3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:38 [ 400 ] DEBUG : Waiting for ClickHouse start in ch3, ip: 172.16.14.7... (cluster.py:2995, start)
2026-10-27 17:46:38 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestsqluserdefinedfunctionsoncluster_ch3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:38 [ 400 ] DEBUG : http://localhost:None "GET /v1.42/containers/7f9e5a263961e35b5b0573f6abed3b61397fa1afc80194d858b045416b7250d4/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:46:38 [ 400 ] DEBUG : ClickHouse ch3 started (cluster.py:2999, start)
------------------------------ Captured log call -------------------------------
2026-10-27 17:46:38 [ 400 ] DEBUG : Executing query SELECT test_function(1); on ch1 (cluster.py:3495, query_and_get_error)
--------------------------- Captured stderr teardown ---------------------------
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
---------------------------- Captured log teardown -----------------------------
2026-10-27 17:46:39 [ 400 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/.env', '--project-name', 'roottestsqluserdefinedfunctionsoncluster', '--file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch1/docker-compose.yml', '--file', '/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch3/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:105, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Stderr:Stopping roottestsqluserdefinedfunctionsoncluster_ch2_1  ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Stderr:Stopping roottestsqluserdefinedfunctionsoncluster_ch3_1  ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Stderr:Stopping roottestsqluserdefinedfunctionsoncluster_ch1_1  ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Stderr:Stopping roottestsqluserdefinedfunctionsoncluster_zoo3_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Stderr:Stopping roottestsqluserdefinedfunctionsoncluster_zoo1_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Stderr:Stopping roottestsqluserdefinedfunctionsoncluster_zoo2_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Stderr:Stopping roottestsqluserdefinedfunctionsoncluster_ch1_1  ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Stderr:Stopping roottestsqluserdefinedfunctionsoncluster_ch3_1  ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Stderr:Stopping roottestsqluserdefinedfunctionsoncluster_ch2_1  ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Stderr:Stopping roottestsqluserdefinedfunctionsoncluster_zoo2_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Stderr:Stopping roottestsqluserdefinedfunctionsoncluster_zoo1_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Stderr:Stopping roottestsqluserdefinedfunctionsoncluster_zoo3_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch1/logs/stderr.log* || true'] (cluster.py:105, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch2/logs/stderr.log* || true'] (cluster.py:105, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch3/logs/stderr.log* || true'] (cluster.py:105, run_and_check)
2026-10-27 17:46:42 [ 400 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/.env', '--project-name', 'roottestsqluserdefinedfunctionsoncluster', '--file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch1/docker-compose.yml', '--file', '/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_sql_user_defined_functions_on_cluster/_instances_0/ch3/docker-compose.yml', 'down', '--volumes'] (cluster.py:105, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing roottestsqluserdefinedfunctionsoncluster_ch2_1  ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing roottestsqluserdefinedfunctionsoncluster_ch3_1  ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing roottestsqluserdefinedfunctionsoncluster_ch1_1  ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing roottestsqluserdefinedfunctionsoncluster_zoo3_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing roottestsqluserdefinedfunctionsoncluster_zoo1_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing roottestsqluserdefinedfunctionsoncluster_zoo2_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing roottestsqluserdefinedfunctionsoncluster_zoo3_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing roottestsqluserdefinedfunctionsoncluster_zoo2_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing roottestsqluserdefinedfunctionsoncluster_ch1_1  ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing roottestsqluserdefinedfunctionsoncluster_ch3_1  ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing roottestsqluserdefinedfunctionsoncluster_zoo1_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing roottestsqluserdefinedfunctionsoncluster_ch2_1  ... done (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stderr:Removing network roottestsqluserdefinedfunctionsoncluster_default (cluster.py:115, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Cleanup called (cluster.py:754, cleanup)
2026-10-27 17:46:43 [ 400 ] DEBUG : !!! Docker info: Client: Docker Engine - Community
 Version:    29.9.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 38
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b7a25261e029d68978bbf290df0a62fc819b3c51
 runc version: v1.5.2-0-g29dd3dc2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-198-generic
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.33GiB
 Name: 02b61c2481ca
 ID: cbb4a713-cdad-4d08-9907-2668eda59ee0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  65.108.242.32:5000
  127.0.0.0/8
 Registry Mirrors:
  http://65.108.242.32:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 24

 (cluster.py:722, print_all_docker_pieces)
2026-10-27 17:46:43 [ 400 ] DEBUG : Docker networks for project roottestsqluserdefinedfunctionsoncluster are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:733, print_all_docker_pieces)
2026-10-27 17:46:43 [ 400 ] DEBUG : Docker containers for project roottestsqluserdefinedfunctionsoncluster are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:741, print_all_docker_pieces)
2026-10-27 17:46:43 [ 400 ] DEBUG : Docker volumes for project roottestsqluserdefinedfunctionsoncluster are DRIVER    VOLUME NAME
 (cluster.py:749, print_all_docker_pieces)
2026-10-27 17:46:43 [ 400 ] DEBUG : Command:docker container list --all --filter name='^/roottestsqluserdefinedfunctionsoncluster_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup)
2026-10-27 17:46:43 [ 400 ] DEBUG : No running containers for project: roottestsqluserdefinedfunctionsoncluster (cluster.py:782, cleanup)
2026-10-27 17:46:43 [ 400 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup)
2026-10-27 17:46:43 [ 400 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup)
2026-10-27 17:46:43 [ 400 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Images pruned (cluster.py:807, cleanup)
2026-10-27 17:46:43 [ 400 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup)
2026-10-27 17:46:43 [ 400 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check)
2026-10-27 17:46:43 [ 400 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check)
=============================== warnings summary ===============================
test_storage_mongodb/test.py::test_arrays[False]
test_storage_mongodb/test.py::test_simple_insert_select[False]
  /ClickHouse/tests/integration/helpers/cluster.py:2452: DeprecationWarning: database_names is deprecated. Use list_database_names instead.
    logging.debug(f"Connected to Mongo dbs: {connection.database_names()}")

test_storage_mongodb/test.py::test_arrays[False]
  /ClickHouse/tests/integration/test_storage_mongodb/test.py:113: DeprecationWarning: add_user is deprecated and will be removed in PyMongo 4.0. Use db.command with createUser or updateUser instead
    db.add_user("root", "clickhouse")

test_storage_mongodb/test.py::test_arrays[False]
test_storage_mongodb/test.py::test_simple_insert_select[False]
  /usr/local/lib/python3.10/dist-packages/pymongo/database.py:1247: DeprecationWarning: Creating a user with the read_only option or without roles is deprecated in MongoDB >= 2.6
    warnings.warn("Creating a user with the read_only option "

test_storage_mongodb/test.py::test_auth_source[False]
  /ClickHouse/tests/integration/test_storage_mongodb/test.py:391: DeprecationWarning: add_user is deprecated and will be removed in PyMongo 4.0. Use db.command with createUser or updateUser instead
    admin_db.add_user(

test_storage_mongodb/test.py::test_complex_data_type[False]
  /ClickHouse/tests/integration/test_storage_mongodb/test.py:262: DeprecationWarning: add_user is deprecated and will be removed in PyMongo 4.0. Use db.command with createUser or updateUser instead
    db.add_user("root", "clickhouse")

test_storage_mongodb/test.py::test_incorrect_data_type[False]
  /ClickHouse/tests/integration/test_storage_mongodb/test.py:292: DeprecationWarning: add_user is deprecated and will be removed in PyMongo 4.0. Use db.command with createUser or updateUser instead
    db.add_user("root", "clickhouse")

test_storage_mongodb/test.py::test_missing_columns[False]
  /ClickHouse/tests/integration/test_storage_mongodb/test.py:424: DeprecationWarning: add_user is deprecated and will be removed in PyMongo 4.0. Use db.command with createUser or updateUser instead
    db.add_user("root", "clickhouse")

test_storage_mongodb/test.py::test_predefined_connection_configuration[False]
  /ClickHouse/tests/integration/test_storage_mongodb/test.py:353: DeprecationWarning: add_user is deprecated and will be removed in PyMongo 4.0. Use db.command with createUser or updateUser instead
    db.add_user("root", "clickhouse")

test_storage_mongodb/test.py::test_secure_connection[True]
  /ClickHouse/tests/integration/test_storage_mongodb/test.py:323: DeprecationWarning: add_user is deprecated and will be removed in PyMongo 4.0. Use db.command with createUser or updateUser instead
    db.add_user("root", "clickhouse")

test_storage_mongodb/test.py::test_simple_insert_select[False]
  /ClickHouse/tests/integration/test_storage_mongodb/test.py:447: DeprecationWarning: add_user is deprecated and will be removed in PyMongo 4.0. Use db.command with createUser or updateUser instead
    db.add_user("root", "clickhouse")

test_storage_mongodb/test.py::test_simple_select[False]
  /ClickHouse/tests/integration/test_storage_mongodb/test.py:49: DeprecationWarning: add_user is deprecated and will be removed in PyMongo 4.0. Use db.command with createUser or updateUser instead
    db.add_user("root", "clickhouse")

test_storage_mongodb/test.py::test_simple_select_from_view[False]
  /ClickHouse/tests/integration/test_storage_mongodb/test.py:79: DeprecationWarning: add_user is deprecated and will be removed in PyMongo 4.0. Use db.command with createUser or updateUser instead
    db.add_user("root", "clickhouse")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================== slowest durations ===============================
135.88s setup    test_storage_s3/test.py::test_ast_auth_headers
117.48s call     test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_no_kdc
91.57s call     test_storage_delta/test.py::test_checkpoint
68.09s setup    test_s3_table_functions/test.py::test_s3_table_functions
64.64s call     test_s3_table_functions/test.py::test_s3_table_functions_timeouts
49.25s call     test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_request_new_ticket_after_expiration
41.63s setup    test_storage_mongodb/test.py::test_secure_connection[True]
30.28s call     test_storage_mongodb/test.py::test_secure_connection[True]
28.34s teardown test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_request_new_ticket_after_expiration
23.30s setup    test_storage_kerberized_kafka/test.py::test_kafka_json_as_string
23.24s teardown test_s3_table_functions/test.py::test_s3_table_functions_timeouts
23.20s call     test_role/test.py::test_roles_cache
22.90s teardown test_storage_delta/test.py::test_types
18.62s call     test_storage_meilisearch/test.py::test_meilimatch[False]
18.05s setup    test_storage_delta/test.py::test_checkpoint
17.12s call     test_storage_meilisearch/test.py::test_meilimatch_secure[False]
16.47s setup    test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1]
14.53s call     test_storage_meilisearch/test.py::test_insert[False]
13.59s call     test_storage_meilisearch/test.py::test_insert_secure[False]
12.47s setup    test_storage_mongodb/test.py::test_simple_insert_select[False]
11.13s call     test_role/test.py::test_introspection
10.82s teardown test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach
10.61s call     test_role/test.py::test_role_expiration[True]
10.29s setup    test_storage_mongodb/test.py::test_arrays[False]
9.74s call     test_settings_profile/test.py::test_settings_from_granted_role
9.36s call     test_storage_mongodb/test.py::test_arrays[False]
9.34s call     test_role/test.py::test_role_expiration[False]
9.19s setup    test_storage_meilisearch/test.py::test_incorrect_data_type[False]
9.16s setup    test_sql_user_defined_functions_on_cluster/test.py::test_sql_user_defined_functions_on_cluster
9.08s call     test_role/test.py::test_revoke_requires_admin_option
8.61s setup    test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach
8.00s call     test_storage_kerberized_kafka/test.py::test_kafka_json_as_string
7.69s call     test_storage_meilisearch/test.py::test_security_levels[False]
6.85s call     test_settings_profile/test.py::test_smoke
6.84s teardown test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6]
6.30s setup    test_select_access_rights/test_main.py::test_alias
6.06s call     test_settings_profile/test.py::test_allow_introspection
5.85s teardown test_select_access_rights/test_main.py::test_select_with_row_policy
5.81s call     test_storage_delta/test.py::test_types
5.80s call     test_storage_delta/test.py::test_multiple_log_files
5.61s call     test_storage_meilisearch/test.py::test_types[False]
5.28s setup    test_settings_profile/test.py::test_allow_ddl
5.17s teardown test_role/test.py::test_set_role
5.17s call     test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1]
5.10s call     test_storage_meilisearch/test.py::test_named_collection_secure[False]
5.10s setup    test_role/test.py::test_admin_option
5.01s teardown test_sql_user_defined_functions_on_cluster/test.py::test_sql_user_defined_functions_on_cluster
4.87s teardown test_settings_profile/test.py::test_smoke
4.74s call     test_storage_meilisearch/test.py::test_types_in_table_function[False]
4.63s teardown test_storage_meilisearch/test.py::test_types_in_table_function[False]
4.10s call     test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5]
4.09s call     test_select_access_rights/test_main.py::test_select_with_row_policy
3.88s call     test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2]
3.88s call     test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3]
3.87s call     test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4]
3.85s call     test_role/test.py::test_admin_option
3.66s teardown test_storage_mongodb/test.py::test_simple_select_from_view[False]
3.64s call     test_role/test.py::test_combine_privileges
3.48s call     test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6]
3.48s setup    test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_request_new_ticket_after_expiration
3.31s call     test_settings_profile/test.py::test_inheritance
3.23s call     test_settings_profile/test.py::test_alter_and_drop
3.18s call     test_storage_meilisearch/test.py::test_simple_select_secure[False]
3.16s call     test_storage_delta/test.py::test_single_log_file
3.15s call     test_storage_delta/test.py::test_metadata
3.09s call     test_select_access_rights/test_main.py::test_materialized_columns
3.01s call     test_select_access_rights/test_main.py::test_select_prewhere
3.01s call     test_select_access_rights/test_main.py::test_select_where
2.88s call     test_select_access_rights/test_main.py::test_select_all_columns
2.88s call     test_settings_profile/test.py::test_function_current_profiles
2.87s call     test_storage_meilisearch/test.py::test_simple_select[False]
2.84s call     test_role/test.py::test_grant_role_to_role
2.80s call     test_storage_delta/test.py::test_partition_by
2.76s call     test_role/test.py::test_function_current_roles
2.67s call     test_select_access_rights/test_main.py::test_select_union
2.61s call     test_role/test.py::test_create_role
2.58s call     test_settings_profile/test.py::test_changeable_in_readonly
2.35s call     test_select_access_rights/test_main.py::test_alias_columns
2.23s call     test_storage_mongodb/test.py::test_simple_insert_select[False]
2.14s call     test_settings_profile/test.py::test_allow_ddl
2.01s call     test_storage_meilisearch/test.py::test_incorrect_data_type[False]
1.93s call     test_storage_mongodb/test.py::test_auth_source[False]
1.88s call     test_select_access_rights/test_main.py::test_select_single_column
1.83s call     test_select_access_rights/test_main.py::test_select_single_column_with_table_grant
1.74s call     test_storage_mongodb/test.py::test_complex_data_type[False]
1.72s call     test_storage_mongodb/test.py::test_incorrect_data_type[False]
1.63s call     test_storage_mongodb/test.py::test_simple_select_from_view[False]
1.62s call     test_storage_meilisearch/test.py::test_table_function_secure[False]
1.51s call     test_storage_meilisearch/test.py::test_named_collection[False]
1.49s call     test_select_access_rights/test_main.py::test_select_all_columns_with_table_grant
1.48s call     test_settings_profile/test.py::test_show_profiles
1.45s call     test_storage_meilisearch/test.py::test_incorrect_data_type_secure[False]
1.42s call     test_role/test.py::test_changing_default_roles_affects_new_sessions_only
1.37s call     test_storage_mongodb/test.py::test_missing_columns[False]
1.30s call     test_storage_mongodb/test.py::test_simple_select[False]
1.29s call     test_select_access_rights/test_main.py::test_alias
1.21s call     test_settings_profile/test.py::test_settings_aliases
1.21s call     test_select_access_rights/test_main.py::test_select_count
1.19s teardown test_settings_profile/test.py::test_allow_ddl
1.10s call     test_settings_profile/test.py::test_changing_default_profiles_affects_new_sessions_only
1.10s teardown test_settings_profile/test.py::test_alter_and_drop
0.97s teardown test_settings_profile/test.py::test_allow_introspection
0.96s teardown test_select_access_rights/test_main.py::test_alias
0.96s call     test_storage_mongodb/test.py::test_predefined_connection_configuration[False]
0.91s call     test_storage_meilisearch/test.py::test_table_function[False]
0.90s teardown test_settings_profile/test.py::test_inheritance
0.90s teardown test_settings_profile/test.py::test_set_profile
0.89s call     test_s3_table_functions/test.py::test_s3_table_functions
0.86s call     test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach
0.82s teardown test_settings_profile/test.py::test_changing_default_profiles_affects_new_sessions_only
0.81s call     test_select_access_rights/test_main.py::test_select_join
0.80s teardown test_settings_profile/test.py::test_show_profiles
0.80s teardown test_settings_profile/test.py::test_changeable_in_readonly
0.80s teardown test_role/test.py::test_create_role
0.77s teardown test_settings_profile/test.py::test_settings_from_granted_role
0.77s teardown test_role/test.py::test_admin_option
0.76s teardown test_settings_profile/test.py::test_settings_aliases
0.75s teardown test_select_access_rights/test_main.py::test_select_join
0.74s call     test_storage_mongodb/test.py::test_no_credentials[False]
0.73s call     test_role/test.py::test_set_role
0.71s teardown test_settings_profile/test.py::test_function_current_profiles
0.70s teardown test_select_access_rights/test_main.py::test_select_all_columns
0.70s teardown test_role/test.py::test_grant_role_to_role
0.66s teardown test_select_access_rights/test_main.py::test_alias_columns
0.64s teardown test_select_access_rights/test_main.py::test_select_prewhere
0.63s teardown test_select_access_rights/test_main.py::test_materialized_columns
0.63s teardown test_role/test.py::test_combine_privileges
0.59s teardown test_select_access_rights/test_main.py::test_select_count
0.59s teardown test_select_access_rights/test_main.py::test_select_union
0.58s teardown test_select_access_rights/test_main.py::test_select_where
0.58s teardown test_role/test.py::test_changing_default_roles_affects_new_sessions_only
0.56s teardown test_select_access_rights/test_main.py::test_select_single_column_with_table_grant
0.55s teardown test_select_access_rights/test_main.py::test_select_single_column
0.54s teardown test_role/test.py::test_function_current_roles
0.54s teardown test_select_access_rights/test_main.py::test_select_all_columns_with_table_grant
0.53s teardown test_role/test.py::test_introspection
0.49s teardown test_role/test.py::test_revoke_requires_admin_option
0.49s teardown test_role/test.py::test_role_expiration[False]
0.47s setup    test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_no_kdc
0.44s teardown test_role/test.py::test_role_expiration[True]
0.43s teardown test_role/test.py::test_roles_cache
0.43s call     test_settings_profile/test.py::test_set_profile
0.37s setup    test_select_access_rights/test_main.py::test_select_prewhere
0.36s setup    test_select_access_rights/test_main.py::test_materialized_columns
0.33s setup    test_select_access_rights/test_main.py::test_select_all_columns_with_table_grant
0.32s setup    test_select_access_rights/test_main.py::test_select_union
0.29s setup    test_select_access_rights/test_main.py::test_select_with_row_policy
0.28s setup    test_select_access_rights/test_main.py::test_select_join
0.28s setup    test_select_access_rights/test_main.py::test_select_all_columns
0.28s setup    test_select_access_rights/test_main.py::test_select_where
0.28s setup    test_select_access_rights/test_main.py::test_select_count
0.28s setup    test_select_access_rights/test_main.py::test_alias_columns
0.27s setup    test_select_access_rights/test_main.py::test_select_single_column
0.27s setup    test_select_access_rights/test_main.py::test_select_single_column_with_table_grant
0.27s call     test_sql_user_defined_functions_on_cluster/test.py::test_sql_user_defined_functions_on_cluster
0.01s setup    test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2]
0.00s teardown test_storage_delta/test.py::test_checkpoint
0.00s teardown test_storage_mongodb/test.py::test_simple_insert_select[False]
0.00s setup    test_storage_meilisearch/test.py::test_simple_select_secure[False]
0.00s teardown test_storage_s3/test.py::test_multipart[negative]
0.00s setup    test_role/test.py::test_role_expiration[False]
0.00s teardown test_storage_mongodb/test.py::test_arrays[False]
0.00s setup    test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4]
0.00s teardown test_storage_s3/test.py::test_ast_auth_headers
0.00s teardown test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1]
0.00s teardown test_storage_meilisearch/test.py::test_meilimatch_secure[False]
0.00s setup    test_settings_profile/test.py::test_settings_from_granted_role
0.00s teardown test_storage_delta/test.py::test_multiple_log_files
0.00s teardown test_storage_mongodb/test.py::test_secure_connection[True]
0.00s setup    test_role/test.py::test_introspection
0.00s teardown test_s3_table_functions/test.py::test_s3_table_functions
0.00s setup    test_settings_profile/test.py::test_function_current_profiles
0.00s setup    test_settings_profile/test.py::test_settings_aliases
0.00s teardown test_storage_kerberized_kafka/test.py::test_kafka_json_as_string
0.00s teardown test_storage_meilisearch/test.py::test_meilimatch[False]
0.00s setup    test_storage_meilisearch/test.py::test_simple_select[False]
0.00s setup    test_role/test.py::test_role_expiration[True]
0.00s setup    test_settings_profile/test.py::test_changeable_in_readonly
0.00s setup    test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5]
0.00s setup    test_storage_s3/test.py::test_insert_with_path_with_globs
0.00s setup    test_storage_s3/test.py::test_get_file_with_special[space]
0.00s setup    test_storage_meilisearch/test.py::test_meilimatch[False]
0.00s setup    test_storage_s3/test.py::test_create_new_files_on_insert
0.00s setup    test_settings_profile/test.py::test_set_profile
0.00s teardown test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_no_kdc
0.00s setup    test_storage_mongodb/test.py::test_missing_columns[False]
0.00s setup    test_storage_s3/test.py::test_format_detection
0.00s setup    test_storage_s3/test.py::test_environment_credentials
0.00s setup    test_storage_meilisearch/test.py::test_insert[False]
0.00s setup    test_role/test.py::test_set_role
0.00s setup    test_settings_profile/test.py::test_inheritance
0.00s setup    test_storage_s3/test.py::test_get_file_with_special[plus]
0.00s teardown test_storage_delta/test.py::test_single_log_file
0.00s setup    test_storage_mongodb/test.py::test_simple_select_from_view[False]
0.00s setup    test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6]
0.00s teardown test_storage_s3/test.py::test_insert_with_path_with_globs
0.00s setup    test_storage_meilisearch/test.py::test_types[False]
0.00s setup    test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3]
0.00s setup    test_role/test.py::test_changing_default_roles_affects_new_sessions_only
0.00s setup    test_settings_profile/test.py::test_smoke
0.00s setup    test_storage_s3/test.py::test_filtering_by_file_or_path
0.00s setup    test_storage_s3/test.py::test_get_path_with_special[plus]
0.00s setup    test_storage_mongodb/test.py::test_simple_select[False]
0.00s teardown test_storage_mongodb/test.py::test_simple_select[False]
0.00s setup    test_role/test.py::test_grant_role_to_role
0.00s setup    test_settings_profile/test.py::test_alter_and_drop
0.00s setup    test_role/test.py::test_roles_cache
0.00s teardown test_storage_s3/test.py::test_empty_put[minio]
0.00s setup    test_storage_s3/test.py::test_multipart[negative]
0.00s teardown test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4]
0.00s teardown test_storage_s3/test.py::test_insert_select_schema_inference
0.00s setup    test_storage_s3/test.py::test_insert_select_schema_inference
0.00s setup    test_storage_delta/test.py::test_metadata
0.00s teardown test_storage_meilisearch/test.py::test_insert[False]
0.00s setup    test_storage_meilisearch/test.py::test_types_in_table_function[False]
0.00s setup    test_role/test.py::test_function_current_roles
0.00s setup    test_storage_mongodb/test.py::test_no_credentials[False]
0.00s teardown test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2]
0.00s setup    test_storage_mongodb/test.py::test_auth_source[False]
0.00s setup    test_settings_profile/test.py::test_changing_default_profiles_affects_new_sessions_only
0.00s teardown test_storage_meilisearch/test.py::test_incorrect_data_type[False]
0.00s setup    test_storage_meilisearch/test.py::test_insert_secure[False]
0.00s setup    test_s3_table_functions/test.py::test_s3_table_functions_timeouts
0.00s setup    test_role/test.py::test_revoke_requires_admin_option
0.00s setup    test_storage_meilisearch/test.py::test_named_collection[False]
0.00s setup    test_storage_meilisearch/test.py::test_meilimatch_secure[False]
0.00s setup    test_role/test.py::test_create_role
0.00s teardown test_storage_meilisearch/test.py::test_insert_secure[False]
0.00s setup    test_storage_mongodb/test.py::test_predefined_connection_configuration[False]
0.00s setup    test_storage_meilisearch/test.py::test_table_function_secure[False]
0.00s teardown test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3]
0.00s setup    test_storage_delta/test.py::test_partition_by
0.00s teardown test_storage_s3/test.py::test_environment_credentials
0.00s teardown test_storage_meilisearch/test.py::test_security_levels[False]
0.00s teardown test_storage_s3/test.py::test_empty_file
0.00s teardown test_storage_s3/test.py::test_infinite_redirect
0.00s teardown test_storage_meilisearch/test.py::test_types[False]
0.00s setup    test_storage_s3/test.py::test_get_path_with_special[space]
0.00s teardown test_storage_meilisearch/test.py::test_simple_select_secure[False]
0.00s setup    test_storage_delta/test.py::test_types
0.00s teardown test_storage_s3/test.py::test_custom_auth_headers_exclusion
0.00s setup    test_settings_profile/test.py::test_allow_introspection
0.00s teardown test_storage_s3/test.py::test_filtering_by_file_or_path
0.00s teardown test_storage_s3/test.py::test_create_new_files_on_insert
0.00s teardown test_storage_mongodb/test.py::test_missing_columns[False]
0.00s teardown test_storage_s3/test.py::test_get_file_with_special[plus]
0.00s setup    test_storage_s3/test.py::test_empty_file
0.00s teardown test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5]
0.00s teardown test_storage_s3/test.py::test_custom_auth_headers
0.00s setup    test_storage_meilisearch/test.py::test_security_levels[False]
0.00s teardown test_storage_delta/test.py::test_partition_by
0.00s setup    test_storage_s3/test.py::test_empty_put[minio]
0.00s setup    test_role/test.py::test_combine_privileges
0.00s setup    test_storage_s3/test.py::test_get_path_with_special[plus2]
0.00s setup    test_storage_meilisearch/test.py::test_named_collection_secure[False]
0.00s setup    test_storage_s3/test.py::test_infinite_redirect
0.00s setup    test_storage_mongodb/test.py::test_incorrect_data_type[False]
0.00s setup    test_storage_meilisearch/test.py::test_table_function[False]
0.00s teardown test_storage_meilisearch/test.py::test_named_collection[False]
0.00s teardown test_storage_s3/test.py::test_get_path_with_special[plus2]
0.00s teardown test_storage_mongodb/test.py::test_no_credentials[False]
0.00s teardown test_storage_s3/test.py::test_get_path_with_special[plus]
0.00s teardown test_storage_s3/test.py::test_get_file_with_special[space]
0.00s setup    test_settings_profile/test.py::test_show_profiles
0.00s teardown test_storage_mongodb/test.py::test_predefined_connection_configuration[False]
0.00s setup    test_storage_mongodb/test.py::test_complex_data_type[False]
0.00s setup    test_storage_s3/test.py::test_custom_auth_headers_exclusion
0.00s teardown test_storage_meilisearch/test.py::test_incorrect_data_type_secure[False]
0.00s teardown test_storage_delta/test.py::test_metadata
0.00s setup    test_storage_s3/test.py::test_custom_auth_headers
0.00s teardown test_storage_mongodb/test.py::test_incorrect_data_type[False]
0.00s teardown test_storage_meilisearch/test.py::test_named_collection_secure[False]
0.00s teardown test_storage_s3/test.py::test_format_detection
0.00s teardown test_storage_s3/test.py::test_get_path_with_special[space]
0.00s setup    test_storage_meilisearch/test.py::test_incorrect_data_type_secure[False]
0.00s teardown test_storage_meilisearch/test.py::test_table_function_secure[False]
0.00s setup    test_storage_delta/test.py::test_multiple_log_files
0.00s teardown test_storage_mongodb/test.py::test_complex_data_type[False]
0.00s setup    test_storage_delta/test.py::test_single_log_file
0.00s teardown test_storage_mongodb/test.py::test_auth_source[False]
0.00s teardown test_storage_meilisearch/test.py::test_simple_select[False]
0.00s teardown test_storage_meilisearch/test.py::test_table_function[False]
=========================== short test summary info ============================
FAILED test_select_access_rights/test_main.py::test_alias_columns - assert "i...
FAILED test_select_access_rights/test_main.py::test_select_count - helpers.cl...
FAILED test_select_access_rights/test_main.py::test_select_join - assert "it'...
FAILED test_settings_profile/test.py::test_show_profiles - AssertionError: as...
FAILED test_storage_mongodb/test.py::test_secure_connection[True] - pymongo.e...
FAILED test_sql_user_defined_functions_on_cluster/test.py::test_sql_user_defined_functions_on_cluster
ERROR test_storage_s3/test.py::test_ast_auth_headers - AssertionError: Expect...
ERROR test_storage_s3/test.py::test_create_new_files_on_insert - AssertionErr...
ERROR test_storage_s3/test.py::test_custom_auth_headers - AssertionError: Exp...
ERROR test_storage_s3/test.py::test_custom_auth_headers_exclusion - Assertion...
ERROR test_storage_s3/test.py::test_empty_file - AssertionError: Expected "OK...
ERROR test_storage_s3/test.py::test_empty_put[minio] - AssertionError: Expect...
ERROR test_storage_s3/test.py::test_environment_credentials - AssertionError:...
ERROR test_storage_s3/test.py::test_filtering_by_file_or_path - AssertionErro...
ERROR test_storage_s3/test.py::test_format_detection - AssertionError: Expect...
ERROR test_storage_s3/test.py::test_get_file_with_special[plus] - AssertionEr...
ERROR test_storage_s3/test.py::test_get_file_with_special[space] - AssertionE...
ERROR test_storage_s3/test.py::test_get_path_with_special[plus2] - AssertionE...
ERROR test_storage_s3/test.py::test_get_path_with_special[plus] - AssertionEr...
ERROR test_storage_s3/test.py::test_get_path_with_special[space] - AssertionE...
ERROR test_storage_s3/test.py::test_infinite_redirect - AssertionError: Expec...
ERROR test_storage_s3/test.py::test_insert_select_schema_inference - Assertio...
ERROR test_storage_s3/test.py::test_insert_with_path_with_globs - AssertionEr...
ERROR test_storage_s3/test.py::test_multipart[negative] - AssertionError: Exp...
PASSED test_settings_profile/test.py::test_allow_ddl
PASSED test_select_access_rights/test_main.py::test_alias
PASSED test_role/test.py::test_admin_option
PASSED test_role/test.py::test_changing_default_roles_affects_new_sessions_only
PASSED test_storage_meilisearch/test.py::test_incorrect_data_type[False]
PASSED test_storage_meilisearch/test.py::test_incorrect_data_type_secure[False]
PASSED test_settings_profile/test.py::test_allow_introspection
PASSED test_role/test.py::test_combine_privileges
PASSED test_select_access_rights/test_main.py::test_materialized_columns
PASSED test_role/test.py::test_create_role
PASSED test_settings_profile/test.py::test_alter_and_drop
PASSED test_select_access_rights/test_main.py::test_select_all_columns
PASSED test_storage_mongodb/test.py::test_arrays[False]
PASSED test_storage_mongodb/test.py::test_auth_source[False]
PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1]
PASSED test_select_access_rights/test_main.py::test_select_all_columns_with_table_grant
PASSED test_role/test.py::test_function_current_roles
PASSED test_settings_profile/test.py::test_changeable_in_readonly
PASSED test_storage_mongodb/test.py::test_complex_data_type[False]
PASSED test_settings_profile/test.py::test_changing_default_profiles_affects_new_sessions_only
PASSED test_storage_mongodb/test.py::test_incorrect_data_type[False]
PASSED test_role/test.py::test_grant_role_to_role
PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2]
PASSED test_storage_mongodb/test.py::test_missing_columns[False]
PASSED test_storage_mongodb/test.py::test_no_credentials[False]
PASSED test_storage_meilisearch/test.py::test_insert[False]
PASSED test_storage_mongodb/test.py::test_predefined_connection_configuration[False]
PASSED test_settings_profile/test.py::test_function_current_profiles
PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3]
PASSED test_select_access_rights/test_main.py::test_select_prewhere
PASSED test_storage_kerberized_kafka/test.py::test_kafka_json_as_string
PASSED test_settings_profile/test.py::test_inheritance
PASSED test_select_access_rights/test_main.py::test_select_single_column
PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4]
PASSED test_settings_profile/test.py::test_set_profile
PASSED test_select_access_rights/test_main.py::test_select_single_column_with_table_grant
PASSED test_settings_profile/test.py::test_settings_aliases
PASSED test_role/test.py::test_introspection
PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5]
PASSED test_select_access_rights/test_main.py::test_select_union
PASSED test_storage_meilisearch/test.py::test_insert_secure[False]
PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6]
PASSED test_select_access_rights/test_main.py::test_select_where
PASSED test_settings_profile/test.py::test_settings_from_granted_role
PASSED test_role/test.py::test_revoke_requires_admin_option
PASSED test_select_access_rights/test_main.py::test_select_with_row_policy
PASSED test_settings_profile/test.py::test_smoke
PASSED test_role/test.py::test_role_expiration[False]
PASSED test_storage_meilisearch/test.py::test_meilimatch[False]
PASSED test_role/test.py::test_role_expiration[True]
PASSED test_s3_table_functions/test.py::test_s3_table_functions
PASSED test_storage_meilisearch/test.py::test_meilimatch_secure[False]
PASSED test_storage_meilisearch/test.py::test_named_collection[False]
PASSED test_storage_meilisearch/test.py::test_named_collection_secure[False]
PASSED test_storage_meilisearch/test.py::test_security_levels[False]
PASSED test_role/test.py::test_roles_cache
PASSED test_role/test.py::test_set_role
PASSED test_storage_meilisearch/test.py::test_simple_select[False]
PASSED test_storage_meilisearch/test.py::test_simple_select_secure[False]
PASSED test_storage_meilisearch/test.py::test_table_function[False]
PASSED test_storage_meilisearch/test.py::test_table_function_secure[False]
PASSED test_storage_meilisearch/test.py::test_types[False]
PASSED test_storage_delta/test.py::test_checkpoint
PASSED test_storage_meilisearch/test.py::test_types_in_table_function[False]
PASSED test_storage_delta/test.py::test_metadata
PASSED test_storage_mongodb/test.py::test_simple_insert_select[False]
PASSED test_storage_mongodb/test.py::test_simple_select[False]
PASSED test_storage_mongodb/test.py::test_simple_select_from_view[False]
PASSED test_storage_delta/test.py::test_multiple_log_files
PASSED test_storage_delta/test.py::test_partition_by
PASSED test_storage_delta/test.py::test_single_log_file
PASSED test_storage_delta/test.py::test_types
PASSED test_s3_table_functions/test.py::test_s3_table_functions_timeouts
PASSED test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_no_kdc
PASSED test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach
PASSED test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_request_new_ticket_after_expiration
======= 6 failed, 76 passed, 14 warnings, 18 errors in 246.81s (0:04:06) =======
Traceback (most recent call last):
  File "/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration/./runner", line 455, in <module>
    subprocess.check_call(cmd, shell=True)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'docker run   --rm --name clickhouse_integration_tests_4vajc8 --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/src/Server/grpc_protos:/ClickHouse/src/Server/grpc_protos --volume=/run:/run/host:ro --mount type=bind,source=/home/ubuntu/_work/_temp/test/dockerd_volume_dir,target=/var/lib/docker -e DOCKER_HELPER_TAG=1864 -e DOCKER_BASE_TAG=1864 -e DOCKER_KERBERIZED_HADOOP_TAG=1864 -e DOCKER_KERBEROS_KDC_TAG=1864 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=1864 -e DOCKER_MYSQL_JAVA_CLIENT_TAG=1864 -e DOCKER_MYSQL_JS_CLIENT_TAG=1864 -e DOCKER_MYSQL_PHP_CLIENT_TAG=1864 -e DOCKER_NGINX_DAV_TAG=1864 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=1864   -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_NEW_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_role/test.py::test_admin_option test_role/test.py::test_changing_default_roles_affects_new_sessions_only test_role/test.py::test_combine_privileges test_role/test.py::test_create_role test_role/test.py::test_function_current_roles test_role/test.py::test_grant_role_to_role test_role/test.py::test_introspection test_role/test.py::test_revoke_requires_admin_option 'test_role/test.py::test_role_expiration[False]' 'test_role/test.py::test_role_expiration[True]' test_role/test.py::test_roles_cache test_role/test.py::test_set_role test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts test_select_access_rights/test_main.py::test_alias test_select_access_rights/test_main.py::test_alias_columns test_select_access_rights/test_main.py::test_materialized_columns test_select_access_rights/test_main.py::test_select_all_columns test_select_access_rights/test_main.py::test_select_all_columns_with_table_grant test_select_access_rights/test_main.py::test_select_count test_select_access_rights/test_main.py::test_select_join test_select_access_rights/test_main.py::test_select_prewhere test_select_access_rights/test_main.py::test_select_single_column test_select_access_rights/test_main.py::test_select_single_column_with_table_grant test_select_access_rights/test_main.py::test_select_union test_select_access_rights/test_main.py::test_select_where test_select_access_rights/test_main.py::test_select_with_row_policy 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6]' test_settings_profile/test.py::test_allow_ddl test_settings_profile/test.py::test_allow_introspection test_settings_profile/test.py::test_alter_and_drop test_settings_profile/test.py::test_changeable_in_readonly test_settings_profile/test.py::test_changing_default_profiles_affects_new_sessions_only test_settings_profile/test.py::test_function_current_profiles test_settings_profile/test.py::test_inheritance test_settings_profile/test.py::test_set_profile test_settings_profile/test.py::test_settings_aliases test_settings_profile/test.py::test_settings_from_granted_role test_settings_profile/test.py::test_show_profiles test_settings_profile/test.py::test_smoke test_sql_user_defined_functions_on_cluster/test.py::test_sql_user_defined_functions_on_cluster test_storage_delta/test.py::test_checkpoint test_storage_delta/test.py::test_metadata test_storage_delta/test.py::test_multiple_log_files test_storage_delta/test.py::test_partition_by test_storage_delta/test.py::test_single_log_file test_storage_delta/test.py::test_types test_storage_kerberized_kafka/test.py::test_kafka_json_as_string test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_no_kdc test_storage_kerberized_kafka/test.py::test_kafka_json_as_string_request_new_ticket_after_expiration 'test_storage_meilisearch/test.py::test_incorrect_data_type[False]' 'test_storage_meilisearch/test.py::test_incorrect_data_type_secure[False]' 'test_storage_meilisearch/test.py::test_insert[False]' 'test_storage_meilisearch/test.py::test_insert_secure[False]' 'test_storage_meilisearch/test.py::test_meilimatch[False]' 'test_storage_meilisearch/test.py::test_meilimatch_secure[False]' 'test_storage_meilisearch/test.py::test_named_collection[False]' 'test_storage_meilisearch/test.py::test_named_collection_secure[False]' 'test_storage_meilisearch/test.py::test_security_levels[False]' 'test_storage_meilisearch/test.py::test_simple_select[False]' 'test_storage_meilisearch/test.py::test_simple_select_secure[False]' 'test_storage_meilisearch/test.py::test_table_function[False]' 'test_storage_meilisearch/test.py::test_table_function_secure[False]' 'test_storage_meilisearch/test.py::test_types[False]' 'test_storage_meilisearch/test.py::test_types_in_table_function[False]' 'test_storage_mongodb/test.py::test_arrays[False]' 'test_storage_mongodb/test.py::test_auth_source[False]' 'test_storage_mongodb/test.py::test_complex_data_type[False]' 'test_storage_mongodb/test.py::test_incorrect_data_type[False]' 'test_storage_mongodb/test.py::test_missing_columns[False]' 'test_storage_mongodb/test.py::test_no_credentials[False]' 'test_storage_mongodb/test.py::test_predefined_connection_configuration[False]' 'test_storage_mongodb/test.py::test_secure_connection[True]' 'test_storage_mongodb/test.py::test_simple_insert_select[False]' 'test_storage_mongodb/test.py::test_simple_select[False]' 'test_storage_mongodb/test.py::test_simple_select_from_view[False]' test_storage_s3/test.py::test_ast_auth_headers test_storage_s3/test.py::test_create_new_files_on_insert test_storage_s3/test.py::test_custom_auth_headers test_storage_s3/test.py::test_custom_auth_headers_exclusion test_storage_s3/test.py::test_empty_file 'test_storage_s3/test.py::test_empty_put[minio]' test_storage_s3/test.py::test_environment_credentials test_storage_s3/test.py::test_filtering_by_file_or_path test_storage_s3/test.py::test_format_detection 'test_storage_s3/test.py::test_get_file_with_special[plus]' 'test_storage_s3/test.py::test_get_file_with_special[space]' 'test_storage_s3/test.py::test_get_path_with_special[plus2]' 'test_storage_s3/test.py::test_get_path_with_special[plus]' 'test_storage_s3/test.py::test_get_path_with_special[space]' test_storage_s3/test.py::test_infinite_redirect test_storage_s3/test.py::test_insert_select_schema_inference test_storage_s3/test.py::test_insert_with_path_with_globs 'test_storage_s3/test.py::test_multipart[negative]'  -vvv" altinityinfra/integration-tests-runner:1864 ' returned non-zero exit status 1.